ElevenLabs lets you build conversational voice bots with Claude and GPT

If you have ever tried building a conversational AI phone agent, you know the plumbing is a nightmare. You had to stitch together a speech-to-text engine, pipe that into an LLM, send the output to a voice generator, and somehow write custom logic to handle conversational interruptions without crashing.

ElevenLabs is now taking over the whole pipeline. The voice-cloning startup just launched a tool on its developer platform that lets anyone build complete, real-time conversational AI agents.

No more stitching APIs together

ElevenLabs head of growth Sam Sklar told TechCrunch that customers were already hacking their text-to-speech API into conversational bots. But two roadblocks kept ruining the experience: plugging in private knowledge bases and handling callers who interrupt mid-sentence. That led the company to build a turnkey agent builder from scratch.

Inside the dashboard, you can pick a template, define a persona, set the opening greeting, and choose a primary language. Crucially, ElevenLabs is not locking you into one AI provider: you can plug in Google Gemini, OpenAI GPT, or Anthropic Claude, or even hook up your own custom LLM.

Developers can dial in response latency, stability, token usage, and maximum call lengths. Feeding the bot context is as simple as uploading a document, pasting text, or dropping a website URL. The platform also lets companies extract structured data like names and emails, and evaluate whether a call succeeded using plain-language rules.

The feature ships with SDKs for Python, JavaScript, React, and Swift, plus a WebSocket API for deeper control.

Going after Whisper and OpenAI

To make this work, ElevenLabs quietly developed its own speech-to-text (STT) technology to handle the incoming voice. While the company is not selling its STT engine as a standalone API yet, it confirmed it might do so down the road, which would place it in direct competition with OpenAI's Whisper, Deepgram, and AssemblyAI.

The launch comes as ElevenLabs reportedly seeks new funding at a valuation above $3 billion. It enters a crowded arena alongside voice-bot startups like Vapi and Retell, but its primary target is OpenAI's Realtime API. ElevenLabs believes its deep voice controls and model-agnostic flexibility will win over developers.

Why it matters

ElevenLabs is making the jump from being a mere voice component to owning the entire customer-facing runtime.

For builders creating sales agents, inbound support desks, or interactive game characters, this lowers the barrier drastically. Building a voice pipeline that reliably detects when a user speaks up mid-sentence usually requires weeks of painful audio engineering. Now, it is essentially a toggle.

There is, however, a fundamental architectural gamble here. OpenAI is moving toward native end-to-end speech models where latency and inflections are handled inside a single neural net. ElevenLabs is still chaining speech-to-text, an LLM, and text-to-speech together, which inherently risks latency overhead.

Still, we think ElevenLabs made a very smart bet on neutrality. Many enterprise teams do not want to be married exclusively to OpenAI's ecosystem. By letting developers swap between Claude, GPT, and internal weights while tapping into industry-leading voice quality, ElevenLabs just made itself the most practical voice layer on the market.