Skip to content
IFAI

SWE-Bench Pro

Product2 stories

  1. IndustryReach 44

    OpenAI finds 30% of SWE-Bench Pro coding tasks are broken

    OpenAI just published an audit of SWE-Bench Pro, revealing that roughly 30% of its coding tasks suffer from serious evaluation flaws like overly strict tests and misleading prompts. After screening tasks with automated pipelines and software engineers, the lab retracted its earlier recommendation of the benchmark and called for reliable evaluations built specifically for AI models.

  2. ResearchReach 38

    OpenAI ditches SWE-bench Verified over benchmark contamination

    OpenAI just declared SWE-bench Verified unreliable for evaluating frontier coding models and stopped reporting its scores. An audit revealed that 59.4% of problems failed by OpenAI o3 had flawed test cases that rejected valid solutions, while multiple frontier models memorized human patches from training data. Why it matters: When models pass tests by reciting memorized training data, benchmarks stop measuring real-world coding capability.