Ai2 drops MolmoMotion for 3D motion forecasting

MolmoMotion is a new open-source 3D motion forecasting model from Ai2. Feed it an initial video frame, a few 3D points on an object, and a plain-English instruction, and it predicts where those points will move in 3D metric space over the next few seconds.

Most computer vision systems are great at analyzing what already happened on screen. But if you want a robot to pick up a mug or a video generator to respect the laws of physics, the system needs to look forward. That is exactly what MolmoMotion is built to do.

How it works

Instead of burning compute rendering full future video frames, MolmoMotion tracks sparse, object-attached 3D points in world coordinates. This approach is class-agnostic, holds steady across camera angle shifts, and produces trajectories that downstream planners can consume directly.

The system runs on the Molmo 2 backbone. It locates the referenced object, identifies the query points, and rolls out the predicted 3D path. Ai2 released two distinct variants.

MolmoMotion-AR uses autoregressive generation, treating quantized 3D coordinates like text tokens. Because each step conditions on prior coordinates, it delivers the sharpest accuracy on deterministic tasks.

MolmoMotion-FM uses flow matching in continuous 3D space. It turns noise into trajectories, making it much better at handling ambiguity when a prompt allows for multiple realistic motions.

To train both models, Ai2 built an automated pipeline that converted internet clips into MolmoMotion-1M, a dataset featuring 1.16 million trajectory-paired videos spanning 736 motion types and 5,600 distinct objects.

Benchmarks and robot tests

On Ai2's new PointMotionBench benchmark, which covers 2,700 human-validated clips, MolmoMotion beat pixel-space video generators and kinematic baselines across 111 object categories.

The real test comes in downstream applications. When fine-tuned on the real-world robotic dataset DROID, a policy built with MolmoMotion hit a 76.3% success rate on simulated pick-and-place tasks, compared to 56.0% for Molmo 2 alone. On real hardware, it matched the baseline's 12,000-step test error in just about 2,000 steps.

It also works as a trajectory steering wheel for video generation. Feeding MolmoMotion's 3D tracks into models like DaS produced motion that tracked instructions far better than raw text prompts, beating larger video backbones on key motion metrics.

The current catch is density. MolmoMotion trains on eight query points per object, which handles rigid manipulation well but struggles with complex, deformable surfaces.

Ai2 has made the code, model weights, the 1.16M dataset, and PointMotionBench fully open on GitHub and Hugging Face.

Why it matters

If you are building robot manipulation policies or controllable video pipelines, this is worth cloning right away.

Bypassing raw pixel prediction in favor of sparse 3D point tracks is an elegant architectural choice. Video models often hallucinate physics because predicting every future pixel is needlessly hard. MolmoMotion lets the VLM handle scene understanding while delegating dynamics to lightweight metric trajectories.

Keep in mind that eight points will not solve folding clothes or handling fluids. But as a modular waypoint generator for robotic arms or a physics prior for video diffusion, we think it offers one of the cleanest open foundations available today.