UniAudio 2.0 folds speech, music, and sound effects into one model

Producing audio for a creative project usually means juggling three different AI tools: one for realistic voiceovers, another for sound effects, and a third for background music.

UniAudio 2.0 wants to replace the entire rack with a single prompt. The project just open-sourced a unified audio language model that pulls text-to-speech, sound design, song synthesis, and vocal style editing into one autoregressive architecture. The paper, inference code, and model checkpoints are already public on GitHub.

Splitting audio for logic and sound

Audio foundation models usually run into a frustrating trade-off. Representations optimized for high-level understanding tend to produce robotic, muffled output, while codecs built for high-fidelity waveform reconstruction are nearly impossible to steer using text prompts.

UniAudio 2.0 works around this bottleneck using a discrete tokenizer called ReasoningCodec. Instead of treating sound as a single data stream, it factorizes audio into two parallel token sets: reasoning tokens that align with text for semantic planning, and reconstruction tokens packed with acoustic cues to rebuild clean waveforms.

Using this setup, the researchers trained the system across 100B text tokens and 60B audio tokens. Through multi-stage and multi-task training, the single model handles in-domain benchmarks and zero-shot generation across unseen tasks.

From whisper tracks to Punjabi folk

The demo showcase covers an unusually wide spectrum.

On basic speech synthesis, the model handles English, Mandarin, and Cantonese right out of the gate.

For Foley effects, typing in natural descriptions like a crowd cheering or a race car skidding to a halt triggers matching environmental audio tracks.

It also tackles full music generation across niche genres, from energetic Punjabi folk and Arabic dance tracks to glass harp waltzes. Feed it raw lyrics, and the model attempts to sing the lines on beat.

There is also an InstructTTS feature for contextual voice acting: feed in standard dialogue, tell the model to whisper or shout, and it reshapes the acoustic delivery to fit the mood.

To be fair, the outputs are not flawless. Music tracks can sound somewhat compressed, sung lyrics occasionally slide into garbled syllables, and shouted voice lines sometimes sound timid. But having all these modalities coexist inside one cohesive system is still a major technical feat.

Why it matters

For developers and creators, replacing a fragmented audio stack with one checkpoint is a massive operational win. Running separate heavy models for voice, sound effects, and music strains compute and complicates pipelines. Even if UniAudio 2.0 is not ready to master a commercial album, having a unified engine makes rapid prototyping for game audio and video edits vastly more practical.

For the broader AI ecosystem, this model challenges the assumption that audio modalities must remain siloed. While the field has leaned heavily on diffusion models for sound and specialized engines for music, UniAudio 2.0 shows that autoregressive transformers can tackle every layer of sound when fed the right token structure.

We think the real star here is ReasoningCodec. By cleanly separating semantic reasoning from acoustic reconstruction, the authors have mapped out a blueprint for future audio foundation models to scale their understanding without sacrificing waveform fidelity.