OpenAI fixes AI coding benchmarks with SWE-bench Verified

If you have followed the AI coding race over the past year, you have probably heard of SWE-bench. It tests whether LLMs can resolve real GitHub issues from popular open-source repositories, and it quickly became the gold standard for evaluating coding agents.

There was just one glaring problem: the test itself was full of bad questions. On August 13, OpenAI teamed up with the original creators of SWE-bench to launch SWE-bench Verified, a cleaned-up dataset that fixes major grading flaws.

When the benchmark is broken

Software engineering autonomy is a key metric under OpenAI's Preparedness Framework. But while testing models against SWE-bench, researchers realized the benchmark was systematically underestimating what coding agents could actually do.

The culprit came down to three core issues. First, the unit tests were often overly specific or entirely unrelated to the prompt. In one scikit-learn task, the model was asked to fix an ignored parameter, but the hidden test required raising a specific deprecation warning with an exact text string decided during PR discussions. Because the issue text never mentioned it, any reasonable fix would fail.

Second, many issue descriptions pulled from GitHub were hopelessly underspecified, leaving humans and models alike guessing what the original author wanted.

Finally, setting up the development environments was notoriously flaky. Perfectly valid patches would routinely fail simply because dependencies broke in the test harness.

93 engineers filter out the noise

To clean up the dataset, OpenAI recruited 93 experienced Python software developers to manually audit 1,699 random samples from the SWE-bench test set.

Each sample was graded independently by three separate annotators across a four-point severity scale. The team adopted a conservative ensembling approach: if a single annotator flagged a sample as severely underspecified or having unfair tests, it was tossed out.

The audit uncovered an astonishing amount of bad data. About 38.3% of the samples had underspecified problem statements, and 61.1% had unfair unit tests that filtered out valid fixes. In total, 68.3% of the audited samples were rejected.

From the verified pool, the team created SWE-bench Verified, a clean subset of 500 samples that supersedes the original SWE-bench and SWE-bench Lite sets. It features 196 quick fixes taking under 15 minutes and 45 tough tasks taking over an hour. They also rolled out a Docker-based test harness to eliminate flaky runtime errors.

With the unfair penalties removed, model performance jumped noticeably. GPT-4o resolved 33.2% of the tasks, while the open-source scaffold Agentless doubled its score from 16% on the legacy benchmark.

Why it matters

For engineering teams and developers wondering whether AI agents are ready for production, this release provides a much needed reality check. When agents were stuck at 10% to 20% on the old benchmark, it looked like automated issue resolution was still science fiction. The new numbers show that agents are significantly more capable than the old leaderboard suggested, making them worth piloting on real-world maintenance tasks.

For the AI ecosystem, this sets a higher standard for how we evaluate frontier models. Over the last year, teams poured immense compute into optimizing for leaderboards that were essentially grading against broken answer keys. By working directly with the benchmark's creators rather than inventing an in-house vanity metric, OpenAI effectively reset the playing field for everyone.

We think this approach was remarkably smart, though not without trade-offs. Filtering out nearly 70% of tasks by letting any single negative review disqualify a sample is aggressive, and it likely removes some realistic ambiguity that human engineers navigate daily. But in a field crowded with hype and noisy metrics, sacrificing volume for pristine, reproducible ground truth is the right call.