ByteDance's DreamID-Omni generates multi-person talking videos from photos and audio

Few things ruin an AI-generated video faster than watching two avatars speak, only to have character A moving their lips while character B's voice plays.

ByteDance's Intelligent Creation Lab and Tsinghua University introduced DreamID-Omni to fix that exact headache. Detailed in their recent paper (arXiv:2602.12160v1), the system unifies portrait generation, voice synthesis, and multi-character lip-syncing under a single framework.

No more lip-sync identity crises

Traditional talking-head models usually crumble the moment you put two or more people in the same frame. Voices bleed into the wrong faces, and the result looks like an amateur dub gone wrong.

DreamID-Omni tackles this with what researchers call a dual-layer decoupling strategy. At the signal level, it uses synchronous rotary position embeddings to carve out distinct time slots for each character on the timeline. Semantically, it assigns structured captions and unique ID tags to each persona, locking together their face, voice, and dialogue.

In practice, that means you can upload reference portraits and voice clips of celebrities like Anne Hathaway or Will Smith, type out a script, and have them hold a coherent conversation on screen without swapping voices or botching their mouth movements.

From blank canvas to full video editing

The model does not just build videos from scratch. It also handles two other tasks: driving static portraits with pure audio, and editing existing footage. You can take an existing video, swap out an actor's face, change their voice to match a reference sample, and keep all original motions perfectly intact.

To make this work across different scenarios, the team trained the diffusion transformer in three progressive stages, moving from masked video reconstruction to cross-pair feature decoupling and all-around fine-tuning.

On their new IDBench-Omni benchmark of 200 challenging cases, DreamID-Omni reportedly beat commercial systems like Wan2.6 across lip-sync and identity preservation. A panel of 30 professional video creators also rated it highest across seven subjective quality metrics. The team has already set up a GitHub repository and noted that preparations are underway for an open-source release, targeting a v1 drop in March.

Why it matters

For video creators and indie developers, a unified audio-visual model like this changes the math. Crafting a convincing two-person sketch previously meant jumping between separate cloning, rendering, and lip-syncing tools, only to manually fix drift in post-production. If ByteDance delivers an open-source release in March as promised, small studios could generate multi-character scenes directly from static assets and audio snippets.

On the industry side, this marks a shift away from piecemeal avatar tools toward end-to-end multi-agent orchestration. By giving each avatar a distinct temporal and semantic identity, the team solved a structural flaw that plagued diffusion video generators.

We think this architectural decision to treat identity as a bound constraint is clever and practical. The real test will be inference speed and compute costs once the community gets hands-on. Moreover, giving anyone the ability to seamlessly swap faces and voices in existing videos will inevitably raise fresh concerns around deepfakes and consent.