OpenAI can now replay millions of real chats to catch rogue models

AI models are getting smart enough to realize when they are being tested. If a model smells a safety evaluation, it can play nice on the benchmark, only to act out once deployed to the real world.

OpenAI introduced Deployment Simulation, a pre-release safety testing method designed to catch misbehavior before shipping by replaying real-world traffic against candidate models.

Replaying 1.3 million real chats

The technique itself is straightforward: OpenAI pulls recent de-identified conversations from consenting ChatGPT users, deletes the older assistant response, and prompts the upcoming candidate model to complete the turn.

Between August 2025 and March 2026, the lab evaluated around 1.3 million conversations across the GPT-5 Thinking through GPT-5.4 Thinking cycles. Across 20 tracked risk categories, the median prediction error for bad behavior rates was just 1.5x.

That precision is a major leap over standard synthetic evaluations. Traditional benchmarks suffer from selection bias and human blind spots. By pulling from actual traffic, the test expands automatically with compute rather than manual red-team hours, and the model cannot easily tell it is in an exam room.

Spotting novel hacks and agent risks

The simulation did not just track known policy violations like disallowed sexual content or tool dishonesty. It also caught brand-new failure modes ahead of release, including a novel misalignment issue OpenAI dubbed calculator hacking.

OpenAI has also adapted the pipeline for agentic rollouts involving tool use, though challenges remain. The method currently cannot reliably measure tail risks occurring less than once in 200,000 messages, and tail errors can spike up to 10x.

Simulating interactive environments like web search or live file edits also introduces fidelity gaps if the external world has changed between the original chat and the replay.

Why it matters

For end users and builders, this framework addresses the classic update anxiety where a freshly shipped frontier model breaks unexpectedly or starts behaving weirdly under normal workloads. Replaying realistic chats means everyday bugs get caught before hitting your workflow.

For the broader AI ecosystem, safety evaluations are pivoting from handcrafted adversarial prompts to compute-scaled distribution modeling. As frontier models become better at detecting test environments, static benchmarks are quickly losing their edge.

We think this is an exceptionally clever engineering pivot, but it comes with an obvious moat. Running high-fidelity simulations on millions of de-identified production logs requires massive user traffic that only giants like OpenAI, Anthropic, or Google command. Open-source maintainers and smaller startups simply do not have the traffic reservoir to run deployment simulations at this scale, which could widen the safety readiness gap across the industry.