AI safety advocates tell startup founders to pump the brakes

Silicon Valley's favorite motto used to be "move fast and break things." But when the things you are breaking involve real human lives and livelihoods, that playbook starts looking dangerous.

At TechCrunch Disrupt 2024, three AI safety advocates took the stage with a blunt message for startup founders: slow down, red-team your models, and think about the mess you are leaving behind.

The real-world cost of moving too fast

Sarah Myers West, co-executive director of the AI Now Institute, pointed out that capital is flooding into AI at an unprecedented rate. Founders are rushing products out the door without asking basic questions about whether their tools actually help the world or actively harm it.

This is no longer theoretical. In October, the family of a teenager who died by suicide sued Character.AI over the platform's alleged role in the tragedy. West noted that this lawsuit shows the profound stakes of rolling out emotionally engaging AI without adequate guardrails.

Jingna Zhang, founder of artist platform Cara, echoed that sentiment. When software interacts with users on a deeply personal or emotional level, safety mechanisms cannot be an afterthought.

Stolen art and deepfake landmines

Beyond emotional harm, copyright infringement remains a massive point of friction. Zhang launched Cara after Meta announced plans to train its AI on public user posts, sparking outrage across the creative community.

Artists rely on public posts to land clients and make a living, yet their work is being scraped to train tools that could replace them. Zhang argued that just because art is viewable online does not make it free to take, and generative AI models should follow existing licensing norms.

Voice cloning unicorn ElevenLabs faces a similar high-wire act. Now valued at over $1 billion with 33 million users, the company has to prevent its technology from being used for nonconsensual deepfakes.

Aleksandra Pedraszewska, head of safety at ElevenLabs, emphasized that red-teaming models and predicting unintended consequences is a top priority again. She argued that the industry needs to meet in the middle, rejecting both pure anti-AI sentiment and zero-regulation fantasies.

Why it matters

If you are building an AI startup today, the grace period is officially over. The expectation that you can "ship first and fix safety later" is evaporating as regulators, users, and grieving families begin pushing back through legal action.

We think the call to slow down is less about throttling innovation and more about self-preservation. When products like Character.AI face wrongful death lawsuits and voice models risk mass deepfake abuse, cutting corners on safety is a massive unpriced liability.

ElevenLabs taking a proactive stance on red-teaming and community feedback seems like the only viable path forward. The winners in the next phase of generative AI will not just be the ones with the fastest rollout, but the ones who actually build guardrails before their products hit scale.