DeepMind is giving silent AI videos a voice

AI video generation has come a long way in realism, but hit play and you’re still met with dead silence. Watching a sports car drift or an ocean wave crash in total quiet carries an uncanny valley feel that no amount of visual fidelity can shake.

Google DeepMind unveiled Video-to-Audio (V2A), a new technology designed to give silent footage a complete, synchronized voice. By combining raw video pixels with natural language prompts, V2A generates dramatic musical scores, realistic foley effects, and even spoken dialogue tailored to the on-screen action.

Sculpting soundwaves straight from raw pixels

Traditional video scoring requires meticulous timeline adjustments, nudging footsteps and explosions by the millisecond. Existing automated tools often demand detailed textual directions to know what should sound and when.

DeepMind took a different path. V2A analyzes raw pixels directly, meaning text prompts are entirely optional. If you drop in a silent clip without giving it any notes, the model can still figure out what is happening on screen and generate a soundtrack to match.

To build it, DeepMind tested both autoregressive and diffusion-based architectures. The team found that diffusion models yielded far more realistic and convincing results when syncing audio with visual information.

The system starts by compressing the video into an internal representation. Then, a diffusion model refines sound out of random noise, guided step-by-step by the visual input and any optional prompts. Finally, the latent audio is decoded into a standard waveform and stitched directly onto the video.

To sharpen control, researchers trained the model on video, audio, and rich AI-generated annotations, including precise sound descriptions and spoken transcripts. This lets creators steer the output using positive and negative prompts, giving them an easy way to dial in desired instruments or filter out unwanted background noise.

From archival films to the stubborn lip-sync hurdle

The technology is not just for generative models like Google’s Veo. DeepMind demonstrated that V2A can breathe new life into silent archival reels and vintage films. Users can generate an unlimited number of alternative audio tracks for any single clip until they find the right mood.

Still, V2A remains a research project behind closed doors, largely because several technical hitches remain unsolved.

For one, audio output quality is tightly tethered to video input quality. If the source video contains visual artifacts or unfamiliar distortions that fall outside the training data, the generated audio degrades fast.

Speech synchronization is an even tougher challenge. While V2A tries to align generated speech from transcripts with on-screen mouth movements, video generation models often create character faces without knowing what words are being spoken. The resulting mismatch produces jarring, unconvincing lip-syncing that ruins the illusion.

DeepMind also noted that before public access is considered, the tool will undergo strict safety assessments, with all generated audio watermarked via its SynthID technology.

Why it matters

Bridging the gap between silent moving images and full audiovisual production is what will turn AI video from a novelty into a viable creative medium. For solo creators, having an automated Foley artist and composer eliminates hours of tedious library searching and timeline nudging.

We think DeepMind’s decision to condition audio generation directly on raw visual pixels is the right technical bet. Bypassing manual alignment marks a huge leap forward in making generative video genuinely useful.

However, treating video and audio as separate sequential pipelines has clear limits. The lip-sync problem proves that until video models are conditioned on the audio script at the moment of pixel creation, characters will continue to look like dubs over mismatched puppets. Full audiovisual realism still needs a unified model that thinks in sight and sound simultaneously.