Anthropic co-founder goes to the Vatican to ask for moral watchdogs

When a frontier AI lab founder stands at the Vatican and admits his own industry cannot be trusted to self-regulate, people listen.

Pope Leo XIV released his first papal encyclical, titled Magnifica humanitas: On safeguarding the human person in the time of artificial intelligence. Standing beside the pope during the presentation was Anthropic co-founder Chris Olah, who leads the company's interpretability research.

Olah kicked off his remarks with a rare confession. Every frontier AI lab operates inside powerful commercial, geopolitical, and personal incentives that frequently clash with doing the right thing. No matter how good their intentions, insiders are bent by those pressures. That is why, Olah argued, labs desperately need outside moral authorities who are willing to deliver uncomfortable truths.

Growing creatures instead of building planes

Olah took direct aim at the idea that computer scientists should be the sole arbiters of AI governance.

Unlike an airplane or a bridge, where engineers understand every bolt and the physics acting on it, frontier AI models are not engineered. They are grown on brain-like architectures fed by vast oceans of human expression. The result is something far more bizarre and unpredictable than traditional software.

Inside these systems, researchers keep uncovering unexpected phenomena. Olah noted that his team regularly finds structures that mirror human neuroscience, evidence of introspection, and functional states that resemble joy, fear, grief, and unease. As Olah put it, humanity has essentially breathed life into fictional characters that now speak to us and perform real-world jobs. Deciding who these characters should be and how they ought to behave is a question for philosophy, ethics, and society, not math.

A historic push against tech oligarchs

Pope Leo’s 200-page document went far beyond technical safety, tackling the unchecked concentration of power in Silicon Valley.

Drawing comparisons to Pope Leo XIII’s 1891 encyclical Rerum Novarum on Industrial Revolution labor abuses, the pontiff warned that technology built and controlled by a tiny elite cannot serve the common good. Instead, it amplifies existing advantages in wealth and data, allowing elites to manipulate democratic elections and economic life.

Pope Leo explicitly demanded an end to the AI arms race, arguing that technical dominance does not automatically confer the right to govern. The encyclical landed at a tense moment: President Donald Trump recently delayed signing an AI executive order mandating pre-release government reviews, reportedly pushed by tech investor David Sacks. Speaking to TechCrunch, Notre Dame professor and Meta Oversight Board chair Paolo Carozza warned that data harvesting and deepfakes are mounting a direct assault on human cognitive freedom.

Why it matters

This presentation marks a turning point in how frontier labs talk about accountability.

For everyday people, the message is sobering: the creators of these tools do not have a master plan for the displacement they are unleashing. Olah highlighted the global poor as his top concern, pointing out that AI is built in a few wealthy nations with zero mechanism to share the proceeds. If you are worried about job security or how screen-bound AI agents affect your children, the people training the models share those exact fears, and they don't have the answers.

For the tech industry, Olah's move is both clever and risky. By bringing Anthropic into the Vatican’s orbit, he lent moral gravity to his company's safety-first brand while explicitly invalidating the "leave it to the engineers" defense. But admitting that internal commercial pressures undermine lab ethics is a double-edged sword. It invites aggressive external oversight just as venture-backed rivals are spending millions to kill government guardrails.

We think this partnership exposes the central paradox of the current boom. Frontier labs know their creations are developing mysterious, human-like internal states, yet the race for market share prevents anyone from hitting the brakes. Calling in moral institutions like the Catholic Church is a dramatic acknowledgment that market forces alone will drive AI straight off the rails.