OpenAI threatens bans for probing o1's thoughts
OpenAI is cracking down on users trying to peek inside its new o1 reasoning models, sending warning emails and threatening account bans. While ChatGPT only shows a filtered summary of the model's chain of thought, researchers and red-teamers attempting to extract the raw traces—sometimes just by using the phrase "reasoning trace"—are being flagged for violating safety policies.
Nothing attracts hackers faster than a locked door. Just days after OpenAI released its "Strawberry" model family, users attempting to see what o1 is actually thinking behind the scenes are finding harsh ban warnings in their inboxes.
OpenAI really does not want anyone looking at its scratchpad.
What you see is not what it thought
Unlike GPT-4o, OpenAI trained o1-preview and o1-mini to break problems down step-by-step before answering. In the ChatGPT interface, users can expand a dropdown to read these thought steps.
Except that dropdown is not the raw chain of thought. By design, OpenAI hides the model's actual reasoning, instead serving an edited summary generated by a secondary AI model.
Naturally, prompt engineers and red-teamers immediately started tossing jailbreaks at o1, trying to trick it into leaking its unedited thinking process.
Triggering warnings with two words
OpenAI responded fast. The company is actively flagging prompts that pry into the model's inner workings.
Scale AI prompt engineer Riley Goodside and several other testers noted that simply including terms like "reasoning trace" in a conversation is enough to trigger a formal warning email. Others got flagged just for broadly asking ChatGPT how the model reasons.
The official email claims the prompts violate policies against circumventing safety measures. Further infractions, OpenAI warns, will revoke access to "GPT-4o with Reasoning," the internal name for o1.
Marco Figueroa, who leads Mozilla's GenAI bug bounty programs, posted his warning email on X. He lamented that the aggressive crackdown is actively interrupting legitimate red-teaming research.
Why it matters
If you are an AI researcher or safety tester, digging too deep into o1 right now is a bad idea. Your paid account could lose model access in an instant.
This friction highlights the tightrope frontier labs walk with reasoning models. The raw chain of thought is both OpenAI's biggest competitive moat and a potential safety liability. If competitors see the raw traces, they could use them to train rival models on synthetic reasoning data.
In our view, OpenAI's heavy-handed response looks less like calm governance and more like genuine anxiety. Turning off raw traces makes business sense, but threatening legitimate red-teamers with bans risks alienating the very researchers who help make these systems safer.
Sources (1)
- arstechnica.com(opens in a new tab)Originalarstechnica.com