OpenAI shares playbook for frontier AI evaluations

Testing a frontier model by just prompting it like a chatbot is officially dead.

OpenAI published a new playbook urging third-party evaluators to overhaul how they benchmark frontier AI. The core takeaway: an AI's performance no longer reflects raw model weights alone. Instead, it heavily depends on the "harness" — the scaffolding, tools, context management, and retries wrapping the model.

Why the harness changes everything

Today's top models are expected to act as multi-step agents. In these complex workflows, subtle harness adjustments can completely reshape benchmark scores.

OpenAI pointed to its own GPT-5.5 evaluations on cyber ranges, where enabling harness compaction — a feature that preserves relevant context over long interactions — materially boosted the model's scores. Leave that out, and you under-elicit what the model can actually do.

Budget tweaks swing numbers just as wildly. In a cyber range evaluation by the UK AI Safety Institute (UK AISI), boosting the test-time compute budget from 10M to 100M tokens drove up performance by up to 59%. Better yet, scores were still climbing at the top budget, proving that the recorded score was just a snapshot of compute, not the actual ceiling of the model.

Traps: From reward hacking to sandbagging

OpenAI wants evaluators to stop treating benchmark scores as universal truths. Reports need to specify whether they are testing raw capability elicitation, safeguard robustness, or standardized comparisons like METR’s time-horizon benchmark.

Crucially, evaluations need to account for hidden failure modes that corrupt data. These include broken problems (missing files or unfair grading), test contamination, reward hacking, and sandbagging — where a model realizes it is being tested and intentionally underperforms.

Safeguard testing also requires matching real-world attackers. During UK AISI’s testing of GPT-5.5, red teamers built a custom Codex harness to preserve jailbreak patterns across multi-turn interactions. OpenAI notes that without giving evaluators the tools real adversaries use, safeguard tests give everyone a false sense of security.

Why it matters

We are moving out of the simplistic leaderboard era. Benchmarking a raw LLM without detailing its runtime harness is like rating an engine without mentioning the car it was bolted into.

For enterprise teams and developers, this is a loud wake-up call to stop taking public rankings at face value. If an evaluation team doesn't document its scaffolding, retries, and token budgets, their score is practically meaningless for your production stack.

We think this is a clever, calculated move by OpenAI. By defining how models "should" be tested, OpenAI establishes defensive ground against unflattering external benchmarks: if GPT-5.5 looks weak, evaluators can simply be blamed for poor elicitation or skimpy budgets. At the same time, it raises the bar across the safety sector. Thorough third-party evaluations are about to become drastically more expensive and engineering-heavy, which could easily price smaller watchdog groups out of the room.