How OpenAI tripled GPT-5.6 Sol's ARC-AGI-3 score with two API toggles

When OpenAI first saw GPT-5.6 Sol stumble on ARC-AGI-3, the team was genuinely stumped. The model had already tackled long-standing math conjectures and beaten games like Pokémon FireRed and Slay the Spire. Yet faced with simple 2D puzzle games, it scored a miserable 7.8%, while GPT-5.5 scraped together just 0.4%.

ARC-AGI-3 drops AI agents into unfamiliar mini-games without rules or instructions, testing how well they can learn purely through trial and error. Humans average 48% on these puzzles. So why was a frontier model failing so hard?

It turns out the benchmark's testing harness was basically giving the model short-term amnesia after every single move.

Tossing out the scratchpad

To keep comparisons fair across different providers, ARC used a generic harness without any special tool features. But that setup came with two fatal flaws.

First, the harness discarded the model's private reasoning chain immediately after every action. GPT-5.6 Sol could still see what button it had just pressed, but all the logic, plans, and hypotheses leading up to that move vanished into thin air. Every single turn required solving the puzzle from scratch.

Second, the harness used a rolling truncation window. Once conversation logs crossed 175,000 characters, earlier actions were dumped. Not only had the model forgotten what it was thinking, it was rapidly losing track of what it had already tried.

OpenAI swapped the harness over to its modern Responses API and flipped on two settings already running inside ChatGPT and Codex: retained reasoning and compaction. Instead of hard-clipping context, compaction intelligently summarizes older state while passing the previous response ID to preserve thinking chains.

The results were wild. GPT-5.6 Sol's score on the public task set leaped from 13.3% to 38.3%—nearly tripling its performance and closing the gap on the human average. At the same time, because the model no longer needed to re-evaluate the board every turn, it used 6x fewer output tokens.

Why it matters

This isn't just OpenAI showing off a higher benchmark number; it's a massive reality check on how we evaluate and build with AI agents.

For developers building complex, multi-step workflows, the takeaway is crystal clear. Truncating context windows or stripping out internal reasoning chains to save pennies usually backfires, driving token costs up as the model spins its wheels in confusion. OpenAI is telling builders to ditch the legacy Chat Completions API and use context compaction with retained thinking if they want real reasoning performance.

For the broader industry, it exposes a widening rift in AI benchmarking. Evaluators love standardized, generic harnesses to keep tests "objective." But frontier reasoning models are now tightly co-designed with specific context-management pipelines. Testing a model stripped of its memory pipeline doesn't measure raw intelligence—it measures how gracefully a model degrades when blinded.

The catch, of course, is vendor lock-in. Squeezing peak performance out of GPT-5.6 Sol now requires buying entirely into OpenAI's proprietary context management architecture.