Just Dub It translates videos into new languages with matching lip sync

Just Dub It is an open-source tool that handles video dubbing and lip syncing in a single pipeline. Give it a video in one language, and it can re-voice the clip into French, German, Portuguese, or other languages while regenerating realistic lip movements that match the new audio, all while keeping the speaker's original face intact.

The developers published the complete repository on GitHub and Hugging Face, covering everything from inference scripts and LoRA weights to the full training dataset. If you create multilingual content or want to experiment with local audiovisual diffusion, this is worth checking out immediately.

What it does

Traditional video translation is usually a clunky assembly line. You transcribe the audio, translate the text with an LLM, run a text-to-speech voice clone, and finally use a dedicated lip-sync model to warp the speaker's mouth. The results often look unnatural, with jittery teeth and mismatched lighting.

Just Dub It takes a different route through joint audio-visual diffusion. Instead of stitching separate tools together, it generates translated speech and synchronized lip motions simultaneously. In the official demos, a sample English monologue was dubbed into French, German, and Portuguese with convincing lip movements and zero facial distortion.

Crucially, the model preserves the speaker's facial identity and handles real-world movement well. Even if the subject in the video turns their head or moves naturally, the system stays locked onto the face rather than warping the features into an uncanny stranger.

How it works

Rather than training a massive new foundation model from scratch, Just Dub It is built as a lightweight LoRA adapter for LTX-2, an open-source audio-visual model known for native sound support and fast generation.

The system takes the original video as conditioning, reads a prompt describing the speaker and translated dialogue, and diffuses both the target audio and synchronized frames together. To run the full pipeline, you pair the dubbing LoRA with the LTX-2 19B checkpoint, a distilled LoRA, the Gemma 3 text encoder, and a 2x spatial upscaler.

During inference, you can render at a base resolution like 512 by 768 over 30 steps. The spatial upscaler then doubles the dimensions to produce a final 1024 by 1536 video at 25 frames per second.

Getting started

You can clone the code directly from GitHub and grab the model files from Hugging Face. The project includes both a command-line interface and a straightforward Python API.

Because LTX-2 is relatively lightweight compared to many video models, the authors note that it runs on most consumer GPUs. If you have enough system RAM, you can even run inference on a CPU, though it will naturally take longer.

Writing prompts is straightforward. You follow a simple template specifying who is speaking, their accent or target language, and the exact dialogue in quotes. For bulk processing, an LLM can easily handle the script translation before passing the prompts into Just Dub It.

Why it matters

We think Just Dub It is a very clever release because it ditches the brittle multi-stage pipeline of the past. Instead of gluing speech synthesis onto face-warping algorithms, it leverages native audio-visual foundation models so sound and lip geometry evolve together.

For independent creators, localization studios, and cross-border marketing teams, having this capability fully open source is huge. It removes the high per-minute costs and privacy risks of proprietary dubbing platforms, letting teams run batch translation locally or fine-tune on specific voices.

That said, loading a 19B base model along with the Gemma encoder still requires decent technical comfort and capable hardware. It is not yet a one-click mobile app. But as a proof of concept for local, open-source video translation, it raises the bar for what consumer hardware can pull off.