Sony's MMHNet generates synced foley audio for 5-minute videos

Adding foley sound effects to silent video is usually a tedious chore. Match every footstep by hand, and a two-minute clip takes all afternoon. Hand it off to today’s AI tools, and things look great for the first ten seconds—until the audio drift kicks in, desyncing footsteps and turning ambient background noise into a garbled mess.

Sony researchers published a new paper on arXiv detailing MMHNet, a multimodal hierarchical network designed to generate high-fidelity, synchronized audio for videos exceeding five minutes in length.

Training short, generating long with Mamba

Existing video-to-audio tools like Hunyuan Video Foley or MM Audio do a respectable job on brief clips. But stretch them beyond a few dozen seconds, and traditional Transformer attention mechanisms hit a wall: compute costs explode, and the model struggles to link macroscopic scene changes with microscopic frame actions.

Sony addressed this by steering away from standard brute-force architectures. Instead, MMHNet blends a multimodal hierarchical network with a non-causal Mamba framework. Mamba handles the long-range temporal dependencies efficiently, while the hierarchical structure splits broad contextual cues from granular, frame-level events.

Crucially, the team proved a "train-short, test-long" methodology. By training exclusively on short audio-video pairs, the model successfully generalizes to clips lasting more than five minutes during inference without needing massive datasets of uninterrupted long videos.

In benchmark evaluations, MMHNet consistently outperformed previous video-to-audio models. Reviewers and paper metrics highlight its tight sync on transient visual hits—such as the exact frame a ball strikes a table—as well as smooth acoustic shifts during sudden camera cuts.

Why it matters

For animators, indie filmmakers, and video editors, sound design eats up huge chunks of post-production time. An AI engine that reliably scores five-minute sequences without drifting could drastically speed up content pipelines. The catch? Sony has only published the paper for now, with no public weights, code repository, or interactive demo available yet.

From a technical standpoint, Sony's work offers an elegant lesson in architectural efficiency. While much of the AI industry attempts to brute-force long-context problems by training on massive, compute-heavy datasets, MMHNet shows that combining state-space models like Mamba with hierarchical processing can deliver superior results using only short training snippets.

We think this approach bridges an important gap in generative media. Generating convincing visuals is only half the battle; bringing them to life requires sound that reacts to physical collisions in real time. Expect more multimodal researchers to borrow Mamba-style architectures for audio-visual alignment in the months ahead.