Researchers are using Super Mario to benchmark AI

Just when you thought watching AI bumble through Pokémon was the ultimate stress test, researchers at UC San Diego decided to turn up the heat. They dropped today’s leading models straight into a live emulator running Super Mario Bros.

The benchmark, run by UC San Diego’s Hao AI Lab, produced some eye-opening results. Anthropic’s Claude 3.7 took the crown, with Claude 3.5 hot on its heels. Meanwhile, Google’s Gemini 1.5 Pro and OpenAI’s GPT-4o stumbled hard through the Mushroom Kingdom.

Overthinking leads straight into the pit

To be fair, Mario wasn’t running on an ancient NES cartridge. The setup relied on an in-house framework called GamingAgent, which hooked into the game emulator.

GamingAgent fed the AI models live game screenshots alongside basic instructions, like jumping left when an enemy draws near. The models then had to rapidly output Python code to guide Mario through the course.

The team noted that beating levels forced models to devise intricate movement strategies on the fly. But the real shocker came from reasoning-focused powerhouses like OpenAI’s o1.

Despite crushing almost every standard academic benchmark, o1 tanked in Mario, performing noticeably worse than standard non-reasoning models. The culprit? Speed. While o1 sat there burning several seconds of test-time compute to think through its next move, Mario had already plummeted down a bottomless pit or ran face-first into a Goomba.

The evaluation crisis hits the Mushroom Kingdom

Researchers have leaned on games to test AI for decades, but the practice has plenty of skeptics. Games are abstract, mechanically predictable, and offer infinite synthetic training loops, which look nothing like the messy real world.

The spectacle of watching LLMs play retro platformers caught the attention of OpenAI founding member Andrej Karpathy, who took to X to sound the alarm on what he calls an industry-wide evaluation crisis.

Karpathy admitted he has no idea what AI benchmarks to trust anymore, writing that his main takeaway right now is that nobody truly knows how good these frontier models actually are.

Why it matters

This Mario experiment might look like a weekend hackathon stunt, but we think it exposes a massive design trade-off that AI labs have been sweeping under the rug: latency versus reasoning depth.

Over the past few months, the frontier narrative has been dominated by models that deliberate and generate massive chains of thought. That works wonders for coding competitions or legal briefs, but it completely breaks down in interactive, time-critical agent environments. If an autonomous agent needs three seconds to parse an obstacle, the ship has already sailed. Developers building user-facing assistants or robotics pipelines should take this as a clear warning that raw benchmark IQ means nothing if inference speed cannot keep up.

At the same time, it highlights how desperate the industry has become for meaningful evaluations. Standard benchmarks are increasingly saturated and contaminated, driving researchers back to retro video games. But as Karpathy rightly pointed out, gaming metrics still don’t tell us whether an AI can handle genuine real-world ambiguity, meaning the race to measure true machine intelligence remains fundamentally wide open.