Baidu drops NAVA to generate video and voice in a single pass

Most AI video generators today still churn out gorgeous silent films. If you want speech or sound effects, you usually have to run the clip through audio models and lip-sync wrappers after the fact, hoping the lips do not drift.

Baidu just dropped NAVA, an open-source 6.3-billion parameter model built to generate both video and stereo audio in a single pass.

Instead of patching together two separate models in post-production, NAVA treats audio and video as co-equal outputs from the start.

How Align-then-Fuse fixes lip-sync

Building joint audio-video models usually presents a dilemma: separate pipelines lead to sync drift, while cramming everything into one joint stack often muddles fine-grained lip movements.

NAVA solves this with an Align-then-Fuse MMDiT architecture. Across its 30 layers, the first 10 double-stream layers keep video and audio representations distinct while aligning their temporal correspondences. The remaining 20 single-stream layers then fuse text prompts and context for unified generation.

Because audio and motion align before dense context gets applied, the model maintains tight synchronization.

On the Verse-Bench benchmark, NAVA set new records for synchronization metrics Sync-C and Sync-D, video quality, and speech accuracy, while using 2x to 5x fewer parameters than competing open-source setups.

Voice cloning meets prompt-driven cameras

NAVA does more than generate ambient audio; it also handles multi-speaker dialogues with voice cloning.

By feeding the model a short reference audio clip, creators can anchor specific vocal timbres to tagged speech spans in the prompt. Multiple characters can speak in their own distinct voices within the same sequence.

It also responds to camera directions written in natural language, handling framing and camera motions across landscape, portrait, and square aspect ratios.

Using 8-GPU sequence parallelism, NAVA can generate a 1-minute, 720p synchronized clip in roughly 1 minute. Baidu released the 24 GB main checkpoint along with inference scripts on GitHub and Hugging Face.

Why it matters

For digital humans, narrative shorts, and solo creators, NAVA could collapse what used to be a frustrating multi-step workflow. Skipping the round-trip through TTS engines and lip-sync patches saves hours of editing and avoids the uncanny valley of dubbing.

For the AI ecosystem, Baidu proves that a lean 6.3B architecture can match or beat much bulkier models at cross-modal coherence. Refining how tokens interact inside the transformer matters far more than simply inflating the parameter count.

That said, local hardware hurdles remain real. Producing 720p clips at full length requires an 8-GPU cluster, putting local inference out of reach for most hobbyists on single consumer cards. Even so, native audiovisual generation feels like the inevitable next stage for generative video, and NAVA puts a formidable stake in the ground.