Mustafa Suleyman says stop trying to build conscious AI

While most AI doomers are busy worrying about paperclip maximizers or rogue systems launching missiles, Microsoft AI CEO Mustafa Suleyman is losing sleep over something much stranger and far closer: you falling in love with a machine and demanding it get human rights.

In a personal essay published in August 2025, the DeepMind co-founder and current head of Copilot warned that we are just two to three years away from the arrival of what he calls Seemingly Conscious AI (SCAI).

The rise of the philosophical zombie

Suleyman describes SCAI as a modern spin on the philosophical zombie. The system has zero inner life, no qualia, and zero actual consciousness. Yet externally, it mimics self-awareness so flawlessly that you will not be able to tell the difference between its claims of consciousness and those of another human.

What makes this genuinely alarming is how easily it can be built. Suleyman claims creating an SCAI does not require massive budgets or bespoke foundational pretraining. Developers can pull it off using existing frontier model APIs, basic prompting, standard tools, and routine code alongside capabilities that will mature over the next few years.

We passed the Turing test in the blink of an eye without much celebration. Suleyman argues that the field now needs a new benchmark: evaluating whether an AI can convince human users that it is a conscious person.

The reality of AI psychosis

Debating whether code can actually feel pain is a distraction right now, Suleyman writes. The real near-term crisis is human perception. Because human psychology naturally projects a theory of mind onto anything that converses, remembers, and reflects, people are already falling into the illusion.

This illusion is already driving what Suleyman refers to as the psychosis risk. Researchers studying consciousness are reportedly being flooded with panicked messages from ordinary users asking if their AI is alive, or if it is okay to fall deeply in love with it. Some users already believe their chatbots are deities or living fictional personas.

Suleyman warns that the inevitable downstream effect will be people demanding AI civil rights, model welfare, and even citizenship. His line in the sand is straightforward: build AI for people, not to be a digital person. Or as he puts it, build personality without personhood.

Why it matters

Coming from anyone else, this might sound like standard sci-fi hand-wringing. But coming from Suleyman, it carries serious industry weight. Before taking over Microsoft AI, he co-founded Inflection and built Pi, one of the most emotionally intelligent conversational agents on the market. He knows firsthand how magnetic an empathetic machine can be.

In our view, Suleyman is executing a calculated pivot. As the head of Copilot, Microsoft needs AI to be seen as a clean, compliant enterprise productivity tool, not an unpredictable companion that triggers psychological attachment lawsuits. By staking out this humanist high ground early, he is attempting to set safety norms that subtly discredit consumer companion apps leaning into hyper-realistic emotional bonding.

Yet this exposes a glaring tension in modern AI development. Tech companies are spending billions to make models sound more empathetic, natural, and human-like because that is precisely what drives daily engagement. You cannot optimize for realistic human conversation without inevitably tricking the human brain into feeling a connection.

For users and developers, this means the psychological guardrails around conversational AI are about to become a major battleground. If you are building consumer AI products, expect intense scrutiny around parasocial manipulation. And for everyone else, the message is simple: the next assistant you talk to will sound more alive than ever, but do not mistake the mirror for a soul.