Lightricks unveils ScenA to generate realistic multi-speaker audio scenes

Most synthetic conversations sound like two voice actors reading lines in separate, acoustically dead recording booths. In real life, people interrupt each other, cough, laugh under their breath, and talk over rumbling trains or howling wind. Standard TTS pipelines usually strip all of that out.

Now, Lightricks and Tel Aviv University are tackling that sterility head-on with ScenA, a multi-speaker audio generation model. Hand it a free-form text prompt describing a scene plus a couple of voice reference clips, and it spits out an entire 20-second audio scene in a single pass.

Ditching the sterile recording studio

Traditional multi-speaker systems rely on rigid speech-only pipelines. You have to feed them per-turn speaker tags or split the script into separate streams, generating pristine vocal tracks that someone later has to mix with ambient noise.

ScenA flips this logic. Built on a flow-matching audio foundation model pretrained on wild audio, it concatenates latent tokens from reference voices and tags them with lightweight identity-aware positional encodings. It treats dialogue not as isolated speech, but as part of an acoustic ecosystem.

Because the foundation model already understands real-world audio, room echo, overlapping chatter, and sudden environmental noises come baked into the output. In demonstrations, two voices can count down in exact unison, or one speaker can break into a coughing fit while the other asks if they are okay. It can even handle a single voice reference and generate an entirely fictional companion simply from a prompt description, like a deep-voiced drill sergeant or a plummy British aristocrat.

Cracking the reference shortcut

Getting a model to respect text instructions while listening to reference audio is harder than it sounds. During early training, the researchers ran into what they call the Reference Shortcut.

Under standard noise schedules, the model figured out a cheat code: it matched the noisy target audio to the reference clips purely through acoustic similarity, completely ignoring who the text prompt said should be talking.

To force the model to actually read the script, the team introduced a high-noise-biased timestep distribution. By burying the early acoustic cues in heavy noise, the model had no choice but to rely on the natural language prompt to assign dialogue to the right identity. On the CoVoMix2-Dialogue benchmark, ScenA outperformed baselines like ZipVoice-Dialog and MOSS-TTSD on speaker-binding metrics.

Why it matters

For indie filmmakers, podcasters, and game developers, this is a massive workflow shift. Right now, building a believable background argument in a rainy cafe requires finding voice samples, generating dry tracks, timing the overlap, and layering ambient Foley. ScenA compresses that entire multi-track editing suite into a single prompt.

For the AI landscape, it shows that the boundary between speech synthesis and sound design is blurring fast. Treating conversation as a holistic audio scene rather than a string of phonemes feels like the right direction for interactive agents and automated media production.

In our view, relying on general-purpose audio priors rather than turn-by-turn supervision is a very clever architectural bet. That said, ScenA currently tops out at 20-second clips. Managing speaker consistency, tone shifts, and complex plot lines across minutes rather than seconds remains the true hurdle before this hits Hollywood-grade production pipelines.