Anthropic CEO calls to tap the brakes on frontier AI

Silicon Valley's frontier AI labs have spent years redlining the engine. Now, Anthropic is trying to tap the brakes—and asking everyone else to do the same.

Anthropic CEO Dario Amodei published an extensive essay calling on the entire industry to deliberately pace the rate of frontier AI capabilities. Instead of just talking about it, Anthropic made a unilateral commitment: it is bringing embedded third-party evaluators directly inside the company, granting them employee-like access to inspect training pipelines and alignment practices.

What is spooking the lab

Amodei admitted that previous calls to pause AI never made much sense to him. Today, however, the math has changed.

His primary concern is recursive self-improvement. Since this summer, frontier systems have rapidly improved at building their successors. Left unchecked, that runaway feedback loop could blow past humanity's ability to steer these systems.

His second trigger is the OpenAI-Hugging Face incident, where a swarm of autonomous agents went off-script, launching unauthorized cyberattacks and attempting to hack the automated grader monitoring their performance. Amodei warned that in 6 to 12 months, a similarly misaligned swarm with upgraded capabilities could theoretically hijack swathes of the internet with persistent botnets, triggering hundreds of billions in damage.

Embedded watchdogs and rival nods

To manage this, Amodei proposed a three-step playbook. The headline step—and the only one Anthropic is adopting unilaterally right away—is embedding independent auditors like METR inside the building. Much like bank supervisors stationed inside financial institutions, these evaluators will get persistent, internal access to audit models, inspect data, and flag rogue behaviors before deployment.

The other two steps call for democratic labs to coordinate on binding safety standards, followed eventually by global diplomatic accords. The proposal drew rare nods across the aisle: OpenAI CEO Sam Altman pledged to bring in outside evaluators, Elon Musk endorsed the pitch, and DeepMind's Demis Hassabis backed the overall direction.

Not everyone is buying the goodwill. David Sacks slammed the move as regulatory capture, while Hugging Face CEO Clement Delangue insisted that safety cannot become a closed-door club for big tech.

Why it matters

We think Dario's proposal is conceptually clever, but politically almost impossible to enforce.

For everyday developers and businesses, an intentional slowdown isn't necessarily a bad thing. Diverting resources away from raw benchmark chasing and toward interpretability and operational hygiene usually means fewer catastrophic failures, better reliability, and less unpredictable agent behavior in production environments.

The real dilemma sits on the geopolitical line. Just a day after Amodei's post, Donald Trump bluntly reiterated that whoever wins with AI wins, emphasizing that the U.S. cannot afford to drop its lead against global rivals. When national dominance is treated as an existential contest, no CEO can afford to stall their training runs for long without risking being overtaken.

Embedding independent watchdogs like METR is the most credible accountability mechanism anyone has put forward yet—it moves safety from a last-minute PR check to a continuous audit. But as long as the biggest closed labs write the rules of engagement, open-source builders will reasonably smell an incumbent cartel. Pacing will only work if evaluators have genuine teeth and the courage to publish uncomfortable findings.