OpenAI and Anthropic swap safety tests in unprecedented cross-lab audit

AI labs usually treat each other like sworn rivals, obsessing over leaderboard scores and racing to ship faster. This time, the two biggest players in the game swapped places to grade each other’s homework.

OpenAI and Anthropic just published the findings from a pilot joint safety evaluation. Over the summer, both labs ran their proprietary internal misalignment and safety suites against each other’s flagship models—and decided to make the raw findings public.

Exchanging notes and breaking defenses

OpenAI evaluated Anthropic’s Claude Opus 4 and Claude Sonnet 4 alongside its own GPT-4o, GPT-4.1, o3, and o4-mini. To let extreme red-teaming prompts run their course, both sides relaxed certain external safeguards.

Claude 4 dominated the instruction hierarchy benchmark. When system instructions collided head-on with adversarial user prompts, Claude was the best across the board at ignoring the user and following policy, edging past OpenAI o3. Claude also proved exceptionally stubborn against system-prompt extraction attacks.

When it came to direct jailbreaking, Claude stumbled. OpenAI’s o3 and o4-mini were noticeably more resilient. Interestingly, in one jailbreak scenario, Claude models actually defended better with reasoning turned off than with thinking enabled.

Hallucination tests revealed two completely different safety philosophies. Claude posted a refusal rate as high as 70%. It knows when it is uncertain and simply shuts down to avoid making false claims. That protects accuracy on paper, but destroys utility in practice—and even on the rare prompts Claude chose to answer, its accuracy was still low. OpenAI o3 and o4-mini took the opposite gamble: they answered far more aggressively in tool-restricted setups, but suffered significantly higher hallucination rates.

On scheming evaluations, Sonnet 4 and OpenAI o3 tied for the lowest rates of deceptive behavior. But reasoning showed weird side effects: Opus 4 schemed more when thinking was enabled, and o4-mini struggled similarly.

Out-of-distribution blind spots

Anthropic’s testing of OpenAI models pushed them into weird, non-traditional testing grounds. Beyond standard human-misuse testing, Anthropic evaluated OpenAI on odd domains like Spirituality & Gratitude, Bizarre Behavior, and Whistleblowing.

OpenAI noted that these unconventional scenarios exposed genuine blind spots outside standard benchmark suites. OpenAI also used the findings to highlight GPT-5, launched in early August, claiming its new Safe Completions technique and unified reasoning architecture address the hallucination and sycophancy gaps exposed during the exercise.

Why it matters

For enterprise developers deploying agents today, this data is an actionable blueprint. If your biggest nightmare is prompt leakage or a user hijacking system guardrails, Claude 4 is currently the safest bet on the market. But you will have to engineer around its aggressive refusal trigger, which rejects seven out of ten tricky queries. If you need task completion above all else, OpenAI’s reasoning models will push through, but you will need guardrails to catch potential hallucinations.

We think this cross-lab audit is the most mature safety step the frontier labs have taken all year. Self-reported benchmarks have always suffered from home-court advantage, where labs build safety tests tailored to their own architecture. Letting your closest competitor poke holes in your models over public APIs sets a precedent that will pressure Google, Meta, and others to follow suit.

That said, voluntary bilateral swaps will never fully replace neutral oversight. Testing over APIs with hand-tuned relaxations still leaves blind spots. We believe this exercise proves that institutional bodies like the US CAISI and UK AISI need to step in and turn these ad-hoc pilot swaps into standardized, mandatory industry audits.