Stanford's AI Index: machines are running out of tests to beat

Stanford's HAI just dropped its massive 2024 AI Index Report, and the main takeaway is both awe-inspiring and slightly existential: AI has basically beaten humans at almost every standard benchmark we've ever designed.

In fact, researchers are now facing a bizarre new dilemma. AI models are getting so ridiculously good at passing existing exams that the benchmarks themselves are becoming obsolete. Academics are literally scrambling to design harder tests just to figure out where humans still hold an edge.

Running out of tests

If you haven't been keeping score, the takeover happened fast. AI beat human baselines in image classification back in 2015, conquered reading comprehension in 2017, and took visual reasoning in 2020.

By 2023, machines set their sights on elite cognitive challenges. Take the MATH benchmark, which consists of 12,500 grueling competition-level math problems. In 2021, AI could barely muster a 6.9% score. Fast forward to late 2023, and a GPT-4-based model notched an 84.3% success rate, rapidly approaching the human expert baseline of 90%.

Visual commonsense reasoning (VCR) saw a similar jump, hitting 81.6 points against an 85-point human baseline. That means an AI looking at a cat on a table can now logically predict that the cat might jump or that the furniture can hold its weight.

Image generation told the same steep trajectory. In the HEIM evaluation across 12 critical real-world metrics, no single model swept the board, but DALL-E 2 dominated text alignment, while Stable Diffusion variants like Dreamlike Photoreal took the crown for aesthetics and photorealism.

The genius that still lies

Yet for all this synthetic brilliance, these systems remain stubborn fabulists.

The report highlighted that hallucination remains a severe, unsolved bug across frontier LLMs. The infamous blunder involving New York lawyer Steven Schwartz, who got slapped with a $5,000 fine for citing entirely fabricated case law generated by ChatGPT, was just the most public casualty.

On TruthfulQA, a benchmark engineered to expose common human misconceptions across law, health, and finance, GPT-4 scored 0.59. Sure, that is nearly three times better than GPT-2 back in 2021, but it means models still fail over 40% of the time when tricky factual traps are laid out.

And keep in mind: this Stanford snapshot only captures progress through late 2023, before the arrival of 2024 disruptors like Sora, Claude 3, Suno, and Devin.

Why it matters

We have officially arrived at the post-benchmark era of artificial intelligence.

For everyday users and enterprises, the message is crystal clear. For closed-ended, rule-governed tasks, AI already runs laps around the median human. If you are not utilizing these models for draft analysis, code generation, and math-heavy modeling, you are burning valuable hours. At the same time, because TruthfulQA and HaluEval numbers prove that models still hallucinate with supreme confidence, leaving them unsupervised in high-stakes legal, medical, or regulatory pipelines remains corporate malpractice.

For the tech industry, traditional leaderboards have officially lost their shine. Scoring high on legacy academic benchmarks no longer guarantees that a model behaves well inside messy, multi-turn corporate workflows.

We think the AI ecosystem is dealing with a severe evaluation crisis. Model capabilities are exploding exponentially, but our yardsticks are still stuck in the multiple-choice era. The race over the next year will not just be about making models smarter, but about finding entirely new ways to stress-test their actual reasoning limits.