MMPhysVideo teaches AI video models real-world physics

Ask today's top video AI to show a glass falling off a table, and there is a good chance the glass turns into jelly or morphs into thin air before hitting the floor. While video diffusion models can generate breathtaking visuals, they still struggle with the basic laws of physics.

Researchers introduced MMPhysVideo, a new framework engineered to tackle the warping, distortion, and physical inconsistencies that plague existing video generators. Even better for the open-source community: the team announced plans to fully release both the code and datasets.

Giving AI models a real sense of geometry and motion

Most current video models predict upcoming frames purely by reconstructing pixels. Because they only understand how pixel colors shift across time, they have no real concept of mass, spatial geometry, or object collision.

MMPhysVideo changes that by framing video generation as joint multimodal modeling. The researchers took critical perceptual cues—such as semantics, geometry, and spatio-temporal trajectories—and converted them into a unified pseudo-RGB format. This lets standard diffusion models process real-world dynamics directly alongside visual details.

To prevent the physical cues from messing up visual rendering, the framework uses a Bidirectionally Controlled Teacher architecture. Parallel branches process RGB data and physical perception separately, while two zero-initialized control links gradually train the system to align everything at the pixel level.

Crucially, generating videos does not become slower. The team distilled the teacher's physical priors into a single-stream student model through representation alignment, avoiding extra inference latency. They also built MMPhysPipe, an automated pipeline that uses vision-language models to find key physical subjects and extract rich multimodal annotations.

Why it matters

For creators and game developers, the hardest part of using AI video has always been erratic, uncanny glitches. When objects stay rigid during collisions and trajectories look physically believable, AI clips become vastly more practical for production workflows without endless re-rolling.

For the AI landscape, MMPhysVideo challenges the brute-force belief that scaling pixel data alone will magically teach models real-world physics. Feeding explicit geometric and motion cues into the model—then distilling those lessons down to a lightweight network—is a much smarter and cheaper path forward.

We think the real win here is maintaining inference efficiency. Many attempts to inject physics into generative models make rendering prohibitively slow, but MMPhysVideo avoids that tax altogether. How well it handles tricky edge cases like fluids and soft bodies remains to be seen, but open-sourcing the framework will give everyone a chance to test its limits.