OpenAI's GeneBench-Pro puts AI's biological research taste to the test

Writing a bioinformatics script is easy. The hard part is looking at messy sequencing data and deciding whether a weird spike is a scientific breakthrough or just batch effect noise.

That higher-order judgment—what scientists call "research taste"—is precisely what current AI benchmarks fail to measure. On June 30, OpenAI introduced GeneBench-Pro, a research-level evaluation designed to test whether AI agents can navigate ambiguity and make consequential decisions in computational biology.

Synthetic data with ironclad grading

Biology benchmarks usually run into the same headache: real-world historical data is messy, and two different researchers might choose two different, equally valid cutoffs. If a benchmark insists on a single human author's preference, it is arbitrary; if it grades too loosely, an agent making critical methodological errors can still pass.

GeneBench-Pro dodges this trap by generating all 129 questions synthetically. Because OpenAI controls the full causal structure and the data-generating process, they can grade answers deterministically against known targets while ensuring that minor differences in defensible choices still produce passing results.

Each problem drops an agent into an isolated workspace equipped with a prompt, raw data, and standard tools like Python and PLINK 2.0. External reviewers—including graduate students, postdocs, and professors who evaluated 82 of the questions—estimated that a single problem would take a human expert 20 to 40 hours to finish, putting human labor costs in the thousands of dollars.

GPT-5.6 Sol hits 28.7%, widening the gap with open source

The benchmark shows massive leaps in frontier reasoning alongside glaring failure points.

OpenAI's strongest model, GPT-5.6 Sol, achieved a 28.7% pass rate at maximum reasoning effort, climbing to 31.5% with Pro mode. When OpenAI first started working on the original GeneBench, GPT-5 scored below 5%.

Test-time compute proved decisive. At minimum reasoning levels, GPT-5.6 Sol scraped by with single-digit pass rates. Cranked to the top, it solved nearly six times as many questions as GPT-5.2 while burning about two-thirds fewer tokens.

Interestingly, the gap between frontier GPT models and top open-source contenders like GLM 5.2 was far wider on GeneBench-Pro than on standard coding leaderboards. OpenAI noted this suggests many open models are heavily optimized for programming rather than general scientific reasoning under quantitative uncertainty.

Expert reviewers noted that agents frequently choked on data anomalies, like failing to notice ancestry swaps in genetic datasets before charging forward with standard analysis pipelines.

Ten representative questions are now open-sourced on Hugging Face, with a 50-question subset heading to Artificial Analysis for independent verification.

Why it matters

GeneBench-Pro marks a crucial pivot: the industry is finally moving beyond coding assistants to evaluate AI as actual scientific collaborators.

We think OpenAI's approach of using fully synthetic data with known causal graphs is a clever solve for the deterministic grading dilemma. It eliminates the arbitrary grading biases that plague long-horizon agent benchmarks. However, the limitation is that real-world biology often breaks known causal models; whether agents that excel in simulated environments can handle messy, out-of-distribution wet-lab data remains an open question.

For researchers, a 28.7% success rate proves AI cannot run autonomous discovery yet. But at just a few dollars in compute per problem versus thousands in human labor, having an agent suggest workflows and stress-test data anomalies is already commercially compelling. For model builders, the message is blunt: coding benchmarks are no longer enough to prove frontier intelligence.