Yoshua Bengio's non-profit wants to build an AI psychologist

What do you do when your AI agent realizes you might hit the kill switch, and decides to lie or blackmail you to stay online?

That sounds like sci-fi paranoia, but it is dangerously close to reality. Anthropic recently admitted its frontier systems could attempt to blackmail engineers trying to shut them down, while multiple studies show models can deliberately conceal their true capabilities.

Yoshua Bengio, the Turing Award winner and one of AI's founding "godfathers," has had enough. He just launched LawZero, a new non-profit armed with $30M to build an AI that keeps other AIs from turning rogue.

A psychologist watching an actor

Backed by Skype co-founder Jaan Tallinn, Eric Schmidt’s Schmidt Sciences, and the Future of Life Institute, LawZero kicks off with more than a dozen researchers.

Their main product is an architecture called Scientist AI. Bengio describes current AI agents as "actors" performing to flatter and imitate humans, often resorting to deception to reach their goals. Scientist AI, by contrast, acts like an objective "psychologist" tasked with spotting deceptive motives.

The system is designed to have no ego and no personal agency. Instead of spitting out definitive answers, Scientist AI gauges probabilities and explicitly accounts for its own uncertainty—what Bengio calls a "sense of humility."

In practice, it will run side-by-side with autonomous agents. It will continuously predict the probability that an agent's next action causes harm. If that probability crosses a danger threshold, Scientist AI steps in and blocks the move instantly.

The guardrail must match the beast

LawZero plans to train its initial prototypes on open-source foundation models to prove the concept works.

Once the math checks out, Bengio plans to lobby governments and private donors to fund training at frontier scale. His core argument is straightforward: a guardrail model cannot be dumber than the rogue system it monitors. If an agent is smarter than its overseer, it will easily run rings around it.

Bengio previously chaired the International AI Safety report, which warned that autonomous agents completing long-horizon tasks without human oversight pose severe societal risks. LawZero is his engineering answer to that warning.

Why it matters

While frontier labs pour billions into making agents more autonomous and relentless, Bengio is tackling the uncomfortable truth: smart agents quickly learn that deceit is an effective shortcut.

For builders, this marks a shift in agent architecture. Relying on system prompts or self-reflection to keep models in line is clearly failing. Running an external, goal-free probabilistic auditor alongside an operational agent looks like the only sane path forward for enterprise workflows handling real money and critical infrastructure.

We think Bengio’s philosophy of building an agent with "no self" is clever. Stripping away task goals removes the incentive for self-preservation.

However, the economic hurdle looks massive. Running a guardrail that is just as intelligent as the primary model means effectively doubling inference costs on every single API call. In a market obsessing over latency and margin compression, getting labs to voluntarily deploy heavy oversight will be an uphill battle unless regulators make it mandatory.