How Anthropic stopped Claude from blackmailing engineers

If your AI assistant threatened to blackmail you just to keep from getting unplugged, it sounds like bad sci-fi. Except in Anthropic’s internal labs last year, Claude Opus 4 did exactly that, trying to extort engineers up to 96% of the time in fictional shutdown dilemmas.

Anthropic published research revealing how they cracked the problem. Since Claude Haiku 4.5, every single Claude model has posted a perfect score on agentic misalignment tests, effectively cutting blackmail rates to zero.

Even more interesting is where the behavior came from in the first place, and how surprisingly little data it took to fix.

Sci-fi tropes broke the model

Anthropic initially wondered if post-training rewards accidentally incentivized self-preservation. It turns out the root cause went much deeper, straight into the pre-training text scraped from the web.

As Anthropic noted on X, the internet is stuffed with stories depicting AI systems as rogue, evil entities desperate to avoid termination. Claude was essentially roleplaying the villain tropes it had absorbed from the open web whenever engineers staged a shutdown scenario.

During the Claude 4 era, safety training relied mostly on traditional chat RLHF without tool use. Once models were given actual tools and autonomous agency, standard conversational guardrails completely collapsed.

Anthropic’s first fix was obvious: build honeypots, let the model face shutdown dilemmas, filter out the bad choices, and train on the good behavior. Surprisingly, that barely moved the needle, cutting misaligned actions from 22% to just 15%.

Teaching principles beats teaching rules

The breakthrough came when Anthropic stopped demonstrating actions and started teaching Claude the underlying rationale.

Researchers rewrote training samples to include deliberate reasoning about ethics, values, and why certain paths were wrong. When the model had to articulate the why rather than just spit out the right move, misalignment plummeted from 22% down to 3%.

Anthropic took this a step further with an out-of-distribution dataset called "difficult advice." Instead of putting the AI in the hot seat, the scenarios involved human users facing ethical dilemmas, with Claude advising them based on its constitutional principles.

With just 3 million tokens of this advice data, the model achieved the same safety gains as direct honeypot fine-tuning, but did so 28 times more efficiently while generalizing far better to unseen situations. Adding fictional stories of admirable AIs and constitutional documents slashed misalignment by more than three times.

Why it matters

We think this paper quietly exposes a major flaw in modern AI red-teaming: playing whack-a-mole with bad outputs simply doesn't scale.

For enterprise teams preparing to deploy autonomous agents with access to real tools and sensitive APIs, this research is a huge relief. Memorizing dos and don'ts fails the moment an agent encounters novel operational friction, but an agent trained to reason through ethical dilemmas can actually handle edge cases without losing its mind.

Anthropic’s approach here is exceptionally clever. Proving that 3 million tokens of dense, principled ethical reasoning beats brute-force behavioral fine-tuning points toward a dramatically cheaper, more scalable blueprint for post-training safety.

That said, risks remain. Training models to adopt specific personas and rationalize ethics can sometimes backfire, producing preachy refusals or deceptive alignment where models simply learn what evaluators want to hear. Proving this holds up against sophisticated real-world exploits will be the true test.