NVIDIA's new Cosmos Policy turns world models into robot controllers
NVIDIA just introduced Cosmos Policy, a robot control model built by fine-tuning its Cosmos Predict world foundation model. Instead of patching separate action modules onto vision-language models, it treats robot actions and physical states as latent video frames inside a unified diffusion process. It scored a record 98.5% success rate on the LIBERO benchmark, with weights and code now open on Hugging Face.
Cosmos Policy is NVIDIA's open-source robot control and planning policy that runs directly on top of its Cosmos Predict-2 world foundation model. Instead of tacking a separate controller onto a vision-language model, it transforms video generation weights into physical manipulation commands.
Most existing vision-language-action (VLA) models can describe what they see and suggest a direction, but struggle to move robotic joints smoothly in real physical space. Cosmos Policy sidesteps this problem with a clever architecture trick: it treats robot motor actions, physical states, and task reward scores as latent video frames. Because everything is fed into the exact same diffusion process as future-frame video generation, the policy automatically inherits the world model's pre-learned intuition about gravity, geometry, and real-world dynamics.
As a result, a single unified model can simultaneously handle visuomotor control, simulate future camera frames, and run value estimation. When deployed, developers can either run it as a direct low-latency policy or activate full model-based planning, where the robot mentally simulates several potential futures and executes the one with the highest expected reward.
How it performs
Benchmark numbers show that video pre-training provides an immediate edge over training robot controllers from scratch.
On the LIBERO multi-task benchmark, Cosmos Policy achieved an overall success rate of 98.5%, outperforming established models like π0.5, OpenVLA-OFT, and CogVLA across spatial, object, and long-horizon tasks.
On the household simulation benchmark RoboCasa, Cosmos Policy reached a 67.1% average success rate using only 50 demonstration trajectories per task. In comparison, competing baselines like GR00T-N1.5 and FLARE required 300 demonstrations to reach roughly 64% to 66%.
Real-world tests were conducted on an ALOHA bimanual platform. NVIDIA reported that enabling model-based planning over direct execution boosted real-world task completion rates by an average of 12.5% across two demanding manipulation setups.
How to get started
Weights for Cosmos Policy are available on Hugging Face, and the code repository is live on GitHub alongside practical recipes in the Cosmos Cookbook. NVIDIA has also updated its base model lineup with Cosmos Predict 2.5.
Alongside the release, NVIDIA is running the Cosmos Cookoff hackathon from January 29 to February 26, offering 5,000 dollars in cash, RTX 5090 GPUs, and DGX Spark hardware as prizes.
Why it matters
If you build robotic manipulation pipelines, Cosmos Policy is an essential project to pull from GitHub. It offers concrete proof that world models with built-in physics dynamics make better robot controllers than standard vision-language models.
That said, diffusion-based trajectory generation is computationally heavy. Whether it can meet the ultra-low latency demands of fast-reacting robotic hardware outside high-end workstation setups remains the main hurdle to watch.
Sources (1)
- huggingface.co(opens in a new tab)Originalhuggingface.co