OpenAI finds 30% of SWE-Bench Pro coding tasks are broken

What happens when the benchmark everyone is bragging about turns out to be loaded with broken questions?

OpenAI just dropped an audit of SWE-Bench Pro, one of the most popular coding benchmarks used to test frontier AI. The verdict is not pretty: about 30% of its 731 tasks are functionally broken.

Because of that, OpenAI is officially retracting its earlier recommendation that the AI community switch to SWE-Bench Pro, warning that current leaderboards might be giving researchers a completely distorted signal.

The 80% pass rate illusion

Not long ago, OpenAI investigated SWE-bench Verified and discovered major contamination and design flaws. At the time, they encouraged the developer community to move over to SWE-Bench Pro, which was built around longer-horizon and more realistic repo-level tasks.

Frontier models made it look easy. Over just eight months, model pass rates on SWE-Bench Pro's 731 public tasks jumped from 23.3% to 80.3%.

To see if that leap was real, OpenAI built a rigorous QA pipeline. An initial automated filter flagged 286 potentially broken tasks. Then, Codex-based investigator agents dove into the repos, running tests and failure traces, while teams of five experienced software engineers conducted independent reviews on each flagged task.

In the end, they found that 200 to 249 tasks—around 30% of the entire benchmark—failed to provide an accurate measurement of model skills.

Four ways a benchmark breaks

The team grouped the broken tasks into four main buckets:

First, overly strict tests. These test suites mandate a hyper-specific internal implementation not mentioned in the prompt, failing models whose solutions were functionally correct.

Second, underspecified prompts. Crucial requirements enforced by hidden tests were completely omitted and impossible to infer from context.

Third, low-coverage tests. These shallow tests allowed sloppy, incomplete fixes to pass without catching bugs.

Fourth, misleading prompts that actively nudged the model toward the wrong behavior or contradicted the test cases.

OpenAI noted that SWE-Bench Pro pulls directly from real-world GitHub issues and pull requests. But open-source PRs are designed for messy human collaboration, not sterile, objective machine testing. Human unit tests are usually written to validate one specific commit, not to act as a general spec.

Interestingly, human reviewers were even harsher critics than AI agents. Humans flagged low-coverage tests in 9.4% of the benchmark, compared to just 4.1% caught by the Codex agents.

Why it matters

If you are an engineering team picking models based on public coding benchmarks, look closely. A jump from 23% to 80% looks incredible in marketing decks, but when nearly a third of the test suite is flawed, those scores are padded with lucky passes on lazy tests and unfair failures on good code.

For the AI industry, this is an overdue reality check. Scraping raw GitHub PRs to build "realistic" evaluations was a clever shortcut, but modern frontier models have outgrown it. They are now smart enough to exploit benchmark flaws that humans overlooked.

We think OpenAI making this move is a smart, calculated pivot. By publicly discrediting a benchmark their own models dominate, they position themselves as the adult in the room demanding higher safety and capability standards. But it also creates a massive vacuum: until the community builds truly rigorous, human-architected software benchmarks, measuring real-world agentic coding will remain mostly guesswork.