Nvidia drops DreamDojo to teach robots physics from human video

Teaching a robot to pick up a cup without crushing it usually takes weeks of tedious teleoperation and broken hardware.

Nvidia, alongside researchers from UC Berkeley, Stanford, and UT Austin, just launched DreamDojo, a new robot world model designed to teach machines real-world physics simply by watching humans go about their day.

Learning physics from human eyes

Instead of collecting countless hours of expensive robot demonstrations, the team trained DreamDojo on a massive dataset called DreamDojo-HV. It packs 44,000 hours of human egocentric video, making it the largest dataset for world model pretraining so far. Compared to the previous benchmark, it delivers 15 times more duration, 96 times more skills, and 2,000 times more scenes.

The training works in two steps. First, the model observes human video using latent actions to build an intuitive grasp of how the physical world behaves. Next, it fine-tunes that foundation with continuous action data tailored to specific robot hardware.

Thanks to a distillation process, the system runs at 10 frames per second in real time for over a minute. The researchers demonstrated realistic action rollouts across several commercial platforms, including the GR-1, Unitree G1, AgiBot, and YAM humanoid robots.

Why it matters

We think this sidesteps one of the nastiest bottlenecks in robotics today: collecting training data on physical machines is slow, costly, and fragile. Letting models absorb spatial commonsense from human video before they ever touch real hardware is a clever shortcut.

For enterprise builders and robotics startups, the immediate win is simulation. Instead of testing unproven code on expensive factory prototypes where a single crash costs thousands, teams can simulate policies and test edge cases inside a realistic world model.

That said, watching is not doing. A human hand has soft tissue and complex tactile feedback that a rigid metal gripper simply cannot replicate. Translating visual intuition into smooth, force-sensitive control across unpredictable factory floors remains a tall hurdle, and 10 FPS might still feel sluggish for split-second reflexes.

With robotics startups raising $26.5 billion in 2025 and Jensen Huang betting big on manufacturing automation, DreamDojo shows Nvidia wants to own the software brain of the robotics boom, not just the GPUs running it.