Chatbot Arena under fire for helping AI giants game scores

For the past two years, Chatbot Arena has been the ultimate truth machine for AI. Two models enter, a human picks the better answer, and the leaderboard crowns the king. It was supposed to be the one place AI labs couldn't game the system.

Now, a scathing new paper claims the arena's referee was helping tech giants cheat all along.

The secret sandbox for AI giants

Researchers from Cohere, Stanford, MIT, and Ai2 dropped a study accusing LM Arena of giving preferential treatment to giants like Meta, OpenAI, Google, and Amazon. The core allegation: elite labs were allowed to run private, unlisted tests on dozens of model variants, quietly burying the duds and only publishing the highest scores.

Meta allegedly took full advantage of this perk. Between January and March, leading up to the Llama 4 rollout, Meta tested 27 different model variants inside Chatbot Arena. When launch day came, it only made the score of a single top-ranking variant public.

Cohere’s VP of AI research Sara Hooker didn't mince words, calling the setup straight-up gamification. According to Hooker, only a handful of players were even told private testing existed, and they received vastly more testing bandwidth than rivals.

LM Arena co-founder and UC Berkeley professor Ion Stoica fired back, calling the paper full of inaccuracies and questionable analysis. LM Arena stated that its testing is open to anyone willing to submit models, adding that publishing scores for unreleased, private checkpoints would make no sense for the broader community.

Stacked battles and rigged data

Cherry-picking wasn't the only issue flagged across the 2.8 million battles analyzed over five months. The authors also found that top labs enjoyed a much higher battle sampling rate, which let them harvest mountains of user preference data.

According to the researchers, feeding that extra arena data back into development could artificially pump a model's score on Arena Hard by up to 112%. LM Arena pushed back on X, arguing that Arena Hard scores don't correlate directly with Chatbot Arena ranks.

To be fair, the study has a noticeable weak spot. The researchers had to identify anonymous models by prompting them about their origins, which isn't completely foolproof. A Google DeepMind engineer pointed out that while Google did submit 10 models for pre-release testing, its Gemma open-source team only submitted one.

Even so, the pressure worked. LM Arena confirmed it is working on a new sampling algorithm to balance out battle frequencies across all models.

Why it matters

AI benchmarking has reached a breaking point. Traditional academic benchmarks like MMLU have long been compromised by synthetic training data and direct memorization, leaving Chatbot Arena as the last bastion of trusted evaluation. If labs can quietly cherry-pick the best of 27 variants and optimize solely for arena vibes, that leaderboard tells us very little about real-world performance.

For enterprise buyers and developers, this means public rankings should no longer dictate your model choices. Meta already got caught pulling this stunt with Llama 4, hyping an arena-optimized conversational variant while the actual open-weights model performed far worse.

In our view, the timing makes this look especially ugly. LM Arena just announced it is spinning out into a venture-backed commercial company. It cannot court millions from investors while running a closed-door sandbox for Silicon Valley's most powerful labs. Without transparent sampling logs and open audit trails, the referee risks losing the one asset it cannot buy back: trust.