VideoJAM fixes AI video's physics problem with joint motion representations

Generative video has gotten astonishingly good at making photorealistic stills, but the moment a subject starts running or interacting with the world, things tend to fall apart. Limbs warp, physics glitch, and inertia simply disappears.

That problem isn't accidental. Conventional video diffusion models are trained on pixel reconstruction objectives that bias them toward making individual frames look sharp at the expense of dynamic coherence.

A team of researchers has introduced VideoJAM, a framework designed to fix AI video's broken physics by teaching generators to understand appearance and motion together.

Training on motion, not just pixels

The idea behind VideoJAM centers on a joint appearance-motion representation. Instead of treating motion as an afterthought that emerges from pixel stitching, VideoJAM embeds both the video frame and its motion signal into a single joint latent representation using a simple linear projection layer.

The model is then forced to predict both the visual pixels and the underlying dynamics from this shared representation.

Then comes the inference phase. VideoJAM introduces a mechanism called Inner-Guidance. Rather than relying on heavy external condition inputs, it uses the model's own evolving internal motion predictions as dynamic guidance signals to keep trajectories grounded in reality as the frames unfold.

Plugs right into existing setups

Crucially, VideoJAM doesn't require a massive dataset overhaul or ballooning parameter counts. It is designed to drop into existing video generator architectures with minimal code tweaks.

To prove it, the researchers fine-tuned an open DiT-30B base model using the framework. In head-to-head comparisons on benchmark prompts packed with complex physical motion, VideoJAM-30B achieved state-of-the-art motion coherence.

According to the authors, it not only dramatically improved upon its DiT-30B baseline, but also surpassed top proprietary systems like Sora, Kling, and Runway Gen-3 in maintaining motion integrity without sacrificing visual detail.

The research paper is set to appear at ICML 2025, with qualitative samples hosted on the project page.

Why it matters

For filmmakers, game developers, and VFX artists, unpredictable physics have kept AI video relegated to mood boards and b-roll. You cannot use a shot if a character's hand melts through a table mid-gesture. A framework that genuinely enforces dynamic coherence brings AI one step closer to usable production pipelines.

For the AI landscape at large, VideoJAM challenges the brute-force philosophy. Big labs have tried to solve physics primarily by scaling compute and training datasets to astronomical levels. VideoJAM demonstrates that modifying the objective function and leveraging internal motion representations can deliver superior coherence on an existing 30B architecture.

We think this lightweight adaptability is its biggest strength. While it still needs testing on longer clip generations and real-time workflows, giving open-source models a plug-in motion prior could significantly close the gap with proprietary giants.