Lip Forcing hits 31 FPS real-time lip sync

Getting a video character to sync lips with brand-new audio sounds simple enough, until you actually try to build it for live streaming.

Traditional lightweight tools can keep up with the frame rate, but the mouth movements end up looking like floating paper cutouts. Full diffusion models fix the realism, but at a punishing cost: waiting dozens of seconds just to render a single second of footage.

Researchers from KAIST AI and AIPARK unveiled Lip Forcing, the first autoregressive diffusion model specifically built for video-to-video lip synchronization. And it takes dead aim at that speed-versus-fidelity dilemma.

Cutting diffusion down to two steps

The most aggressive move here is trimming an iterative diffusion process down to just two denoising steps.

The team started by training a massive 14B bidirectional teacher model to capture both rich facial details and precise audio alignment. They then distilled that intelligence into two causal student models: a lightweight 1.3B variant and a heavy-duty 14B flagship.

Running on a single Nvidia H100, the 1.3B student clocks in at 31.58 FPS with a first-frame latency of only 0.32 seconds. That is 17.6 times faster than its bidirectional baseline and safely past the 25 FPS threshold required for true real-time video. Meanwhile, the 14B version runs at 15.11 FPS, delivering nearly a 40x speedup over heavyweights like OmniAvatar and 4.7 times the throughput of LatentSync.

Solving the guidance trap

Anyone who has played with diffusion lip sync knows the classic classifier-free guidance (CFG) trade-off. Crank up the guidance scale, and the lips lock tightly to the audio, but the subject's face warps. Dial it down, and the face stays natural while the mouth starts slacking.

Lip Forcing splits the difference mathematically. Instead of using a static guidance scale, the student model executes an unguided first step to preserve facial identity, then applies targeted guidance in step two to snap the lips into sync. A chunk-wise sliding-window attention mechanism keeps memory overhead tiny, caching just 7 frames so it can stream autoregressively without waiting for future video chunks.

Why it matters

We see Lip Forcing as a genuine turning point for real-time interactive avatars.

Until now, virtual assistants, live dubbing tools, and conversational game NPCs have been stuck in an awkward middle ground: either render fast and look robotic, or look gorgeous and suffer unbearable lag. Hitting 31 FPS on a 1.3B diffusion model proves that cinematic facial sync does not need to be locked behind batch rendering pipelines.

Sure, the reported frame rates were clocked on top-tier H100 silicon, so running this locally on a laptop is not happening tomorrow. But the architecture itself is remarkably lean. By proving that a two-step causal schedule can preserve identity without sacrificing synchronization, KAIST and AIPARK have handed the entire AI video space a blueprint for taking heavy diffusion models truly real-time.