Fish Audio drops S2 with word-level voice directing

Directing an AI voice has always felt surprisingly clunky. You either tweak global sliders like "cheerfulness" that make the entire paragraph sound unnaturally hyped, or you suffer through XML-like SSML tags just to get a single breath right.

Fish Audio took a much cleaner path with Fish Audio S2, its second-generation open-source TTS model released on March 12. Instead of tweaking settings panels, you direct the AI exactly like an actor reading a script.

Stage directions right inside the text

The marquee feature in S2 is open-domain inline tags. You drop plain-language instructions in square brackets anywhere in your script, and the model handles the shift on the fly. Write "[soft voice] Take your time," and the delivery changes instantly.

Critically, this operates at the word level. Placement determines timing. If you write "I didn't want to go [whispering] inside," only the final word drops into a whisper. The model reads the instruction immediately before the targeted word and maintains that tone until the next tag or sentence break.

There is no rigid keyword menu either. Trained on over 10 million hours of audio across roughly 80 languages, S2 understands over 15,000 descriptive prompts in multiple languages, whether it is English, Japanese, or Chinese. You can throw nuanced acting cues at it like "[the calm tone of someone who has done this a thousand times]" and watch it adapt.

Fast latency meets fully open code

Beyond emotional range, the underlying engine is built for production. S2 clocks a Real-Time Factor (RTF) of 0.195 with a first-chunk latency of just 100 milliseconds, fast enough to power real-time conversational agents without awkward dead air.

It also supports long-form text and can render multiple distinct speakers within a single generation pass. Pair that with voice cloning, and generating complex, multi-character audiobooks no longer requires cutting and stitching separate audio tracks in post.

Fish Audio is also keeping things open. S2 model weights, fine-tuning scripts, and the inference stack are available on GitHub and Hugging Face, alongside cloud endpoints via the Fish Audio API and web app.

Why it matters

For indie creators, game developers, and narrative writers, S2 removes the most tedious bottleneck in generative audio: post-production. Writers already think in stage directions, and turning those natural notes directly into acoustic inflection makes high-fidelity audio production feel like typing a draft.

For conversational AI, that 100ms first-chunk response time combined with inline sighs, chuckles, and throat-clearing gives voice agents a genuine shot at conversational realism. Static, monotone assistants are quickly becoming legacy tech.

Strategically, open-sourcing the full stack puts immediate pressure on proprietary API vendors charging premium rates for rigid SSML systems. By treating voice nuances as an open-vocabulary prompt problem rather than an audio-engineering setting, Fish Audio has found an elegant UX layer. The real test now is prompt consistency across diverse accents and edge-case descriptions under heavy production loads.