Why your favorite AI benchmarks are basically meaningless

Every time Big Tech drops a new AI model, we get the exact same slide: a massive bar chart showing their system crushing every standardized benchmark in existence. Google claimed Gemini hit 90% on MMLU to beat human experts, and Mark Zuckerberg quickly bragged that Meta's Llama was scoring around 82.

Now, researchers are pulling back the curtain on these leaderboards. A deep investigation by The Markup revealed that many of the most famous AI benchmarks are aging, riddled with low-quality data, and fundamentally broken.

Testing supercomputers with Reddit scrapers

Carnegie Mellon University assistant professor Maarten Sap put it bluntly, stating that the yardsticks used to measure AI are fundamentally broken.

Part of the problem is sheer age. Popular tests like HellaSwag were built back in 2019 to catch absurd mistakes that models five years ago used to make. Rowan Zellers, the lead author behind HellaSwag, noted it was meant to test simple logic like knowing someone swinging a hula hoop doesn't suddenly step out of a car.

On top of that, many datasets were scraped from Reddit, WikiHow, and trivia sites rather than built by credentialed experts. In some cases, companies used Amazon Mechanical Turk gig workers to write questions that evaluate moral and ethical judgment.

There is also the massive issue of data contamination. Princeton professor Arvind Narayanan noted that older benchmarks have likely been ingested by the models during pre-training, turning the tests into simple memorization. Meanwhile, University of Washington linguist Emily M. Bender argued these tests lack construct validity because next-token predictors do not reason, no matter how high their multiple-choice scores look.

Google told The Markup that it acknowledges academic benchmarks are not foolproof and that high scores do not mean AI can replace doctors. Meta and OpenAI chose not to comment.

Why it matters

We think this benchmark illusion creates real danger for end users. When tech giants boast that an LLM passed the US Medical Licensing Exam, regular people naturally assume the bot is qualified to give medical advice. Knowing these tests are mostly parlor tricks reminds everyone not to trust an AI with high-stakes legal, financial, or healthcare decisions.

For the AI industry, the current leaderboard obsession looks increasingly lazy. Companies keep clinging to flawed five-year-old multiple-choice exams because investors and the media demand clean comparative numbers. But as models saturate these datasets, the industry will be forced to ditch static question-and-answer benchmarks in favor of live, multi-step agent evaluations.