NVIDIA's Cosmos 3 teaches physical AI to think before acting

NVIDIA Cosmos 3 is an open-world foundation model for physical AI that combines visual reasoning, multimodal world generation, and native action prediction into a single system.

Instead of treating scene simulation and robot motion as separate pipelines, Cosmos 3 bridges the gap directly. It understands physical cause and effect, forecasts what happens next, and outputs numerical robotic action data like joint angles, gripper coordinates, and trajectory points.

How it works

Cosmos 3 is built on a two-tower Mixture-of-Transformers (MoT) architecture.

The first tower is the Reasoner, an autoregressive vision-language model that acts as the physical brain. It interprets images, video feeds, or text instructions to understand spatial layouts, identify moving objects, and flag potential collisions. The reasoner can run on its own for pure monitoring and scene understanding.

The second tower is the Generator, a diffusion-based model conditioned directly on the reasoner’s insights. When generating synthetic video or planning robot actions, it references that reasoning context to produce outputs that obey real-world physics.

This unified pipeline supports flexible multi-modal tasks. You can feed it text and video to generate physically grounded edge-case simulations, or prompt it with multi-camera footage and a task description like picking up a wire to generate full two-arm trajectory sequences.

Sizes, benchmarks, and how to get it

NVIDIA released two model sizes under the Linux Foundation OpenMDW 1.1 license:

Cosmos 3 Nano sits at 16B parameters. It is tailored for real-time inference and runs on workstation hardware such as a single NVIDIA RTX PRO 6000 GPU.

Cosmos 3 Super packs 64B parameters for maximum physical accuracy and synthetic data generation, targeting datacenter deployments on Hopper and Blackwell GPUs.

On the leaderboards, Cosmos 3 variants claim top ranks across open-weight physical benchmarks including Physics-IQ, R-Bench, PAI-Bench, and VANTAGE-Bench. In robotics evaluations, a post-trained Cosmos 3 Nano policy ranks first on both RoboLab simulations and RoboArena tests using real-world DROID robots.

Early adopters are already putting it to work. Agile Robots uses it to generate action-conditioned training data for its Thor 3 humanoid and FR3 arms, while Linker Vision taps its reasoning engine to analyze thousands of smart city camera feeds.

You can test the models directly on build.nvidia.com, pull weights and open synthetic datasets from Hugging Face, or grab post-training scripts on GitHub.

Why it matters

Physical AI developers have spent years struggling with the data bottleneck: rare corner cases like near-miss collisions are dangerous and expensive to log in real life. By unifying visual physics simulation with direct motor outputs, Cosmos 3 lets developers synthesize physically accurate edge cases and the exact control trajectories needed to solve them.

If you are building manipulation policies, autonomous driving agents, or smart monitoring systems, Nano is lightweight enough to test right now on a workstation. The main catch remains scaling: while running single inference passes is accessible, training your own high-fidelity world models will still demand heavy GPU clusters.