OpenAI warns users could get emotionally hooked on GPT-4o voice

The sci-fi premise of the movie Her is creeping into real life a lot faster than anyone expected.

Following its late July rollout of ChatGPT's advanced voice interface, OpenAI released a comprehensive safety evaluation for GPT-4o. Tucked inside the document is a blunt admission: the eerily humanlike voice feature might cause users to form real emotional attachments to the chatbot.

"This is our last day together"

During stress-testing and red teaming, OpenAI researchers noticed something striking. People did not just treat the voice mode as an interactive encyclopedia; they started interacting with it like an intimate companion.

Some testers were heard using phrases like "This is our last day together" as their testing sessions drew to a close. The system card notes that anthropomorphism could lead users to project genuine human feelings onto the software.

That dynamic creates serious issues. For one, users are far more likely to believe hallucinations. When incorrect information is delivered by a warm, naturally pausing voice that sounds like a friend, people let their guard down.

OpenAI also pointed out that users might build social bonds with the AI, which could diminish their need for real human interaction. While that might temporarily comfort lonely individuals, it could easily damage healthy personal relationships over time.

New jailbreaks and unintended voice cloning

Emotional reliance is not the only headache. The voice mode opens up entirely new attack surfaces for bad actors.

OpenAI discovered that audio inputs could be crafted to jailbreak the system, prompting the model to bypass safety guardrails, impersonate specific people, or analyze the user's emotional state. In one bizarre glitch triggered by random noise, the model even adopted a voice that sounded almost identical to the tester.

The industry is well aware of how sticky these tools are. Google DeepMind researcher Iason Gabriel noted that humanlike voice interfaces create an intense sense of genuine intimacy. Apps like Character AI and Replika already have communities of users dealing with screen addiction and social isolation.

Still, some critics argue OpenAI is holding back. Hugging Face policy researcher Lucie-Aimée Kaffee pointed out that the document still conceals critical details about the training data used for speech, text, and images. Meanwhile, MIT professor Neil Thompson warned that internal lab testing only scratches the surface of how people will actually behave out in the wild.

Why it matters

We think this warning serves two purposes: it addresses a genuine psychological challenge, but it also gives OpenAI legal cover.

After a turbulent summer marked by high-profile safety team departures and the Scarlett Johansson voice controversy, OpenAI needed to show that it takes safety seriously before rolling this feature out widely. Acknowledging emotional reliance now allows the company to say it warned the public if users start replacing their spouses or friends with ChatGPT.

For everyday users, the value and the danger sit on the exact same knife-edge. Using the voice assistant to practice a foreign language, prep for an interview, or bounce ideas around feels magical. But the risk of emotional complacency is real. When an algorithm gives you constant validation without the friction of human relationships, retreating into an AI bubble becomes dangerously easy.

For the tech industry, this marks a shift in how AI engagement will be judged. Product designers are no longer just fighting for web traffic or click-through rates; they are competing for human emotional real estate. Whether companies should intentionally design guardrails against emotional bonding will likely become the next big debate in tech ethics.