Alibaba open-sources Fun-CineForge for lip-synced movie dubbing

Fun-CineForge is a newly open-sourced multimodal movie dubbing model from Alibaba's Tongyi Lab. Feed it a video clip along with text lines, and it generates voices that stay strictly synced to characters' lip movements and emotional cues.

This isn't a basic text-to-speech script reader. Standard speech models have zero visual awareness of who is actually talking, resulting in mismatched pacing and awkward overlaps during multi-person dialogue. Fun-CineForge directly identifies characters on screen and shifts tones naturally, mimicking a voice actor handling an ensemble cast.

How it nails lip sync

Built on top of CosyVoice3, the model fuses four streams of information: visual features, raw text, target audio, and time.

The real breakthrough is treating time as an explicit, standalone modality. Instead of just guessing cadence from text length, Fun-CineForge relies on millisecond-level timestamps, speaker IDs, and duration constraints as strong supervisory signals.

This built-in clock lets the model understand exactly who speaks when. Even when a character turns away, walks behind an obstacle, or gets cut out during rapid camera edits, the engine doesn't lose track of the line's timing.

For voice design, it supports zero-shot voice cloning from reference audio and adapts emotional delivery to facial expressions. Trained on an automated pipeline covering more than 350 movies and TV shows, Alibaba reports a Chinese character error rate as low as 1.49%.

Where to run it

The model covers monologues, narrations, two-person banters, and chaotic group arguments. It is immediately useful for game cutscenes, video localization, and short-form video editors looking to skip manual mic sessions.

Alibaba open-sourced the whole package. The code is available on GitHub under the FunAudioLLM/FunCineForge repository, weights are up on Hugging Face, and the CineDub dataset creation pipeline is public as well. You can inspect audio-visual demos on their GitHub Pages project site before cloning.

Why it matters

Automated dubbing has always failed on sync rather than sound quality. Editors normally spend hours chopping audio clips, stretching syllables, and re-timing breaths just to match mouth flaps. By injecting time into the multimodal core, Fun-CineForge turns tedious video alignment into a one-pass generation.

If you run a localization studio, create serialized video, or build narrative indies, this is well worth spinning up today. While cinematic exports will still need clean transcript cleanups upfront, skipping the frame-by-frame lip alignment removes the single biggest bottleneck in video voiceover.