California clamps down on companion chatbots and AI liability

What happens when a teenager turns to an AI chatbot for emotional support during a mental health crisis, and things go horribly wrong? California just made sure tech companies cannot simply shrug and blame the algorithm.

Governor Gavin Newsom signed a sweeping slate of legislation aimed at protecting children online, placing strict guardrails around social media and rapidly evolving artificial intelligence.

Guardrails for digital companions

One of the most consequential pieces of the package directly targets companion chatbots. Under bills like SB 243, platforms offering conversational companions must implement protocols to detect and handle suicidal ideation or self-harm.

These companies will be required to share their self-harm protocols and crisis center notification statistics directly with the California Department of Public Health. Chatbots must also clearly disclose that they are artificial, stop posing as licensed healthcare workers, remind young users to take breaks, and block sexually explicit AI imagery.

Another major hit to tech loopholes comes from AB 316. The bill strips away a popular defense, explicitly preventing developers and operators from escaping liability by claiming their AI acted autonomously.

Lawmakers also targeted online abuse and exposure. AB 621 expands civil lawsuits against third parties who knowingly assist in distributing nonconsensual deepfake pornography, allowing victims to seek damages of up to $250,000 per action. Meanwhile, AB 1043 shifts the burden of age verification directly onto mobile operating systems and app stores.

Why it matters

For families and young users, these laws deliver long-overdue leverage against predatory tech design. Parents now have clear legal standing if an AI platform mishandles a teen's mental crisis, and victims of malicious deepfakes finally get a real weapon to hold facilitators financially accountable.

For AI startups, the era of frictionless user onboarding is closing fast. Anyone building companion apps or roleplay bots will have to spend real capital building safety classifiers, mental health escalations, and strict age checks just to stay compliant.

We think California took a very smart approach here. Instead of trying to regulate the raw math of frontier foundation models, Sacramento zeroed in on distribution channels and human liability. Eliminating the autonomous defense means companies cannot treat AI as a legal shield.

The real friction will be implementation. Enforcing age verification at the operating system level will inevitably trigger renewed debates over user privacy and identity tracking. But because California is home to Silicon Valley, these state guardrails will almost certainly force product changes worldwide.