Meta denies artificially inflating Llama 4 benchmark scores

Did Meta cheat on the Llama 4 test?

That was the question dominating Reddit and X over the weekend, following wild rumors that the company cooked its latest benchmark numbers. By Monday, Meta rolled out its top AI leadership to extinguish the fire.

Ahmad Al-Dahle, Meta’s VP of generative AI, pushed back hard on X, calling claims that Meta trained its new Llama 4 Maverick and Scout models on benchmark test sets "simply not true."

The leak that sparked the storm

The controversy appears to have started on a Chinese social media platform, where a user claiming to be a former Meta employee said they resigned in protest over the company's internal benchmarking tactics.

The claim alleged that Meta trained Maverick and Scout directly on test sets—the AI equivalent of memorizing the exam answers beforehand. Doing so artificially inflates benchmark scores while leaving the model fragile on real-world tasks.

The rumor caught fire because developers were already reporting underwhelming performance on certain tasks. Worse, independent researchers on X pointed out that Meta had used an unreleased, experimental version of Maverick on the LMSYS LM Arena leaderboard. The publicly downloadable model reportedly behaved quite differently from the arena version.

Al-Dahle conceded that users are currently seeing "mixed quality" across various cloud platforms hosting the models. His defense: Meta rushed the release out the door the minute it was ready, and it will take a few days for partner cloud implementations to stabilize.

Why it matters

For developers eager to build on Llama 4, this is a clear sign to pause before deploying Maverick or Scout into production. If public implementations are still buggy and vary across cloud providers, give it a week for partners to dial in their deployments—and judge the models on your own evals rather than Meta’s charts.

For the broader industry, this episode highlights a growing crisis of trust around AI benchmarks. When even Meta feels the need to submit unreleased, bespoke checkpoints to LM Arena to secure top ranks, public leaderboards lose their utility.

We think Meta walked right into this PR mess by prioritizing speed over polish. Denying data contamination is one thing, but using an unreleased build to game LM Arena while open-source users deal with a visibly weaker local model looks sloppy. If Meta wants to remain the champion of open-weight AI, it needs to treat its community with consistency rather than benchmark marketing.