Kaggle is putting top AI models on the chessboard to test raw reasoning

What happens when you force the world’s smartest chatbots to play chess without any help from actual chess engines?

As it turns out, pure chaos.

Google’s data science platform Kaggle and Google DeepMind launched the Kaggle Game Arena, an AI benchmarking arena designed to evaluate reasoning capabilities through strategic games. To kick things off, Kaggle pitted eight heavyweight models against each other in a three-day chess tournament: OpenAI’s o3 and o4-mini, Google’s Gemini 2.5 Pro and Gemini 2.5 Flash, Anthropic’s Claude Opus 4, xAI’s Grok 4, DeepSeek-R1, and Moonshot’s Kimi k2.

No Stockfish, just raw hallucinations

Unlike traditional computer chess championships, none of these models were allowed to call tools or check external engines like Stockfish. They received raw text prompts representing the board, were given up to 60 minutes per move, and had to figure out legal moves entirely on their own.

If a model attempted an illegal move, it was given three retries before forfeiting the game on the fourth strike. The event featured commentary and recaps from chess heavyweights including Hikaru Nakamura, Levy Rozman, and Magnus Carlsen.

Day one delivered four clean 4-0 sweeps, sending Gemini 2.5 Pro, o4-mini, Grok 4, and o3 into the semifinals after knocking out Claude Opus 4, DeepSeek-R1, Gemini 2.5 Flash, and Kimi k2.

Yet the sweeps masked widespread confusion on the board. According to Chess.com, Kimi k2 lost every single game against o3 in under eight moves simply by burning through all four illegal move retries. Once the models exhausted memorized opening theory, many suffered severe hallucinations, forgetting piece movements or misplacing board coordinates altogether. One bright spot came from o4-mini, which managed two genuine checkmates against DeepSeek-R1.

Alongside the livestreamed bracket, Kaggle plans to run hundreds of behind-the-scenes matches to build a persistent, dynamic leaderboard across multiple strategic titles, including Go and Werewolf.

Why it matters

Watching elite reasoning models stumble over basic rules exposes a fundamental flaw in text-only spatial awareness.

For developers and enthusiasts, this experiment illustrates why LLMs still struggle as autonomous agents without external verification layers. While models can write brilliant essays and code, tracking physical or stateful changes across a 64-square board purely inside their context window remains remarkably fragile.

From a benchmarking perspective, Kaggle's Game Arena is a smart move. Traditional static benchmarks suffer from saturation and data contamination, whereas zero-sum dynamic games provide fresh, moving targets. Google also plans to introduce social deduction games like Werewolf to evaluate deception and coordination.

Still, enterprise buyers should keep their expectations in check. As Constellation Research analyst Holger Mueller pointed out, LLM esports carries massive entertainment value, but chess prowess rarely translates into enterprise ROI. We think the tournament is great theater and a humbling benchmark, proving that true world modeling inside an LLM is still very much a work in progress.