A new leaderboard tests if chain-of-thought actually helps your model

Most AI benchmarks are stuck in a game of memorization, where models crush standard tests simply because they have already seen the answers.

Hugging Face introduced the Open Chain of Thought Leaderboard to test real reasoning instead. Rather than ranking models by absolute accuracy, the new leaderboard measures the marginal gain: accuracy with chain-of-thought prompting minus accuracy without it.

Measuring the boost, not just the score

Knowing an answer doesn't mean a model can actually reason through it. The Open CoT Leaderboard tackles data contamination by focusing on the difference chain-of-thought makes when solving difficult multi-choice questions from LogiQA and the LSAT.

Without chain-of-thought, the baseline is scored using loglikelihood. When chain-of-thought is turned on, the model first generates an explicit step-by-step reasoning trace before selecting its final answer.

To test how models respond to different nudges, the team used two prompting setups: Classic, which uses the familiar prompt to think step by step, and Reflect, which instructs the model to reflect on the problem abstractly before solving it. Combined with greedy decoding, beam search, and sampling, each model generates reasoning traces under six different regimes. The leaderboard then logs the highest accuracy gain.

Small models punch up, but CoT can backfire

The initial run across 30 models delivered a few immediate surprises.

Smaller open weights proved remarkably capable. Models in the 7B range generated effective reasoning traces that significantly improved their scores. In fact, a compact model like Phi-2 saw a larger marginal accuracy boost from chain-of-thought than the much larger Mixtral.

Fine-tuning also made a clear difference. Instruction- and chat-tuned models consistently outperformed their base counterparts, lifting both their baseline accuracy and their ability to gain ground through reasoning traces.

However, chain-of-thought is far from a silver bullet. The benchmark showed that no single prompting strategy or decoding setup works universally. What lifts performance on one task can tank it on another, with reasoning traces sometimes dragging accuracy below the zero-shot baseline.

Why it matters

For developers building LLM agents, this leaderboard is a practical reality check. Generating reasoning traces burns tokens and adds latency, so knowing which models actually get smarter with step-by-step prompts helps builders avoid wasting inference budgets on models that gain nothing from them.

We think measuring the marginal accuracy delta is a clever, lightweight counter to benchmark gaming. While standard leaderboards favor models that memorize test sets, evaluating the delta isolates how well a model actually leverages reasoning structure.

That said, the approach still has blind spots. The current setup relies solely on multiple-choice formats from two legal and logical datasets. More importantly, a model can still write a flawed reasoning trace and stumble into the correct letter choice by accident, an issue the creators acknowledge will require deeper qualitative trace analysis to solve.