OpenAI ditches SWE-bench Verified over benchmark contamination
OpenAI just declared SWE-bench Verified unreliable for evaluating frontier coding models and stopped reporting its scores. An audit revealed that 59.4% of problems failed by OpenAI o3 had flawed test cases that rejected valid solutions, while multiple frontier models memorized human patches from training data. Why it matters: When models pass tests by reciting memorized training data, benchmarks stop measuring real-world coding capability.
The gold standard for evaluating AI coding capabilities just got torched by the very team that helped build it.
OpenAI announced that it will no longer report scores on SWE-bench Verified and is urging the rest of the industry to drop the benchmark too. The postmortem is brutal: the benchmark is plagued with broken tests that reject working code, and frontier models have essentially memorized the answers.
SWE-bench Verified was released in August 2024 to evaluate models on real-world GitHub issues. Over the last six months, frontier scores slowed to a crawl, nudging from 74.9% to 80.9%. That prompted OpenAI to audit 138 problems where OpenAI o3 struggled across 64 runs. What they found explains the plateau.
Tests reject correct code
Out of the 138 audited problems, a staggering 59.4% contained material flaws in test design or problem descriptions that made them virtually impossible to solve.
About 35.5% had overly narrow test cases that enforced specific implementation quirks rather than functional correctness. For example, in a pylint task, the original pull request introduced a helper function called get_annotation, which the unit test imported directly. A model could fix the underlying bug cleanly, but if it did not guess that exact function name, the test suite threw an import error and failed.
Another 18.8% suffered from tests that were too wide. In a sympy task, the original PR addressed three separate bugs, but the benchmark prompt only described one of them. Models frequently implemented the requested fix correctly, only to fail the tests checking for the other two unspoken bugs.
Models caught memorizing gold patches
Bad tests are only half the scandal. The other half is that every major model is cheating.
Because SWE-bench relies on public open-source repos, models encounter the solutions during web pretraining. OpenAI ran an automated red-teaming probe using GPT-5 against GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash Preview. Every single one regurgitated human-written gold patches verbatim.
Claude Opus 4.5 recalled exact four-line diffs complete with original inline comments. Gemini 3 Flash spat out task descriptions and regex fixes when given nothing more than an issue ID. GPT-5.2 even cited Django 4.1 release notes in its chain-of-thought to divine an unmentioned parameter required by flawed unit tests.
In short, high scores on SWE-bench Verified measure how much GitHub commit history a model memorized, not how well it reasons over code. OpenAI is now developing new evaluations and recommends SWE-bench Pro in the interim.
Why it matters
If you build software with AI, this audit confirms what many developers have felt in practice: benchmark hype rarely matches daily experience. A model boasting an 80% solve rate on SWE-bench Verified might just be reciting PR diffs it ingested during pretraining, offering zero advantage when debugging your proprietary codebase.
For the AI industry, this marks the official death of one of its most respected scorecards. As public benchmarks get cannibalized by massive training runs, public leaderboards lose their ability to distinguish true agentic breakthroughs from glorified autocomplete.
We think OpenAI's decision to blow the whistle is both necessary and tactically sharp. By admitting that its own models memorized the dataset, OpenAI neutralizes rivals who might use Verified gains for marketing and pushes the frontier toward private, contamination-proof evaluations like SWE-bench Pro.
Sources (1)
- openai.com(opens in a new tab)Originalopenai.com