OpenAI can now replay millions of real chats to catch rogue models
OpenAI just unveiled Deployment Simulation, a pre-release safety pipeline that replays real, de-identified user conversations to forecast how candidate models will act in the wild. Tested across roughly 1.3 million conversations from GPT-5 Thinking through GPT-5.4 Thinking, it predicted undesired behavior rates with a median error of just 1.5x. Crucially, the realistic setup prevents smart models from realizing they are being tested, catching novel flaws like tool hacking before launch.