InstructAV2AV syncs audio and video edits with a single prompt

Editing video with AI has always had an awkward blind spot: sound.

You could prompt an AI model to swap an actor's jacket or turn daytime into night, but the audio track remained completely oblivious. Editors still had to manually strip the audio, re-synthesize lines or effects in separate tools, and painstakingly realign them in post.

Researchers unveiled InstructAV2AV, an end-to-end framework designed to edit video visuals and accompanying audio simultaneously through natural language instructions.

One prompt, zero masks

The breakthrough behind InstructAV2AV is treating video frames and sound waves as a single, coherent narrative rather than two separate files.

With just a text command, the framework can swap out characters, alter spoken dialogue, insert new objects, or delete unwanted elements along with their corresponding sounds. Crucially, creators do not need to draw precise masks or segment audio tracks; the model figures out what to change on its own while preserving the background visuals and ambient noise.

To make this possible, the team created InsAVE-80K, the first large-scale joint audio-video editing dataset featuring 80,000 paired source-and-target clips.

How it keeps visual and audio in sync

Under the hood, InstructAV2AV leverages generative backbones incorporating components like Ovi, Wan2.2, and T5 to harness rich pre-trained priors.

To avoid hallucinations and maintain temporal consistency, the framework concatenates the source audio-video inputs with noisy latent codes, firmly anchoring the original context. It then deploys a specialized source-instruction gated attention mechanism to strike a balance between executing new prompt instructions and retaining untouched scene elements.

Evaluations across two benchmarks and 11 distinct metrics show that InstructAV2AV consistently outperforms existing disjointed editing approaches.

Why it matters

We see InstructAV2AV as an essential shift from isolated video generators toward truly multimodal post-production tools.

For creators and editors, it points toward a future where dubbing, localization, and pickup shots happen with a single sentence instead of bouncing between Premiere and a digital audio workstation. It removes the friction of manual sync work.

Across the industry, this puts pressure on pure-video generation labs. Delivering stunning visuals without matching acoustic dynamics is rapidly becoming an outdated shortcut; joint diffusion across sight and sound is clearly the real end goal.

Still, it pays to stay grounded. InstructAV2AV is currently an academic framework, and while its InsAVE-80K dataset is a massive leap forward, maintaining photorealistic lip sync and complex acoustic acoustics under hectic motion will require significant compute and rigorous real-world testing before it lands in your favorite NLE timeline.