Nvidia open-sources Lyra 2.0 to turn videos into explorable 3D worlds

Imagine recording a quick video walkthrough of your home on your phone, and turning it into an interactive 3D space where every couch and mug stays strictly where it belongs—even when you walk away and return an hour later.

Nvidia open-sourced Lyra 2.0, a 3D world generation framework designed to do exactly that. It takes camera-controlled scene videos and lifts them into highly consistent, persistent 3D Gaussian splat environments.

Curing AI's spatial forgetting and drift

Building 3D worlds from generative video has always hit a brutal wall: long-horizon consistency.

Existing video generators tend to suffer from spatial forgetting. Once a room drifts out of the model's immediate temporal context window, the AI simply forgets it. When the camera revisits that room later, the model hallucinates new furniture.

At the same time, autoregressive models accumulate tiny synthesis errors over time, causing severe temporal drift that progressively bends walls and warps geometric layouts.

Lyra 2.0 fixes this through a clever split. It tracks per-frame 3D geometry purely for information routing, effectively creating a 3D index that retrieves past viewpoints and grounds correspondences. Visual appearance is left to generative priors.

To counter drift, Nvidia trained the model on self-augmented histories—feeding it its own slightly degraded frames so it learns how to correct trajectory errors on the fly instead of amplifying them. The resulting video feed is lifted into 3D Gaussian splats for real-time rendering.

A training ground built straight from video

Visual consistency is only half the battle. Nvidia's real endgame here is physical AI.

Once a 3D environment is generated, it can be exported directly into Nvidia Isaac Sim—the virtual gym where autonomous robots and mobile platforms train before touching physical floors.

Instead of paying 3D artists to sculpt digital twins or lugging expensive photogrammetry rigs through a warehouse, researchers can turn everyday video footage into an interactive navigation sandbox.

Remarkably, the entire framework weighs in at just 131MB. That makes it light enough to run smoothly across mainstream consumer GPUs without enterprise-grade hardware clusters.

Why it matters

We think Lyra 2.0 is a textbook example of Nvidia leveraging AI tools to feed its core robotics ecosystem.

For developers and robotics startups, this removes one of the most frustrating bottlenecks in physical AI: the shortage of diverse 3D training scenes. Being able to ingest regular videos and export simulation-ready Gaussian splats for Isaac Sim drastically cuts the time and cost required to test navigation algorithms.

Across the industry, this shifts the battleground of video generation. While competitors race for cinematic realism or longer clip duration, Nvidia treats video generation merely as an intermediate step to create interactive 3D assets. Video becomes a compiler for world-building.

There are clear limits, of course. Lyra 2.0 reconstructs static geometry and visual fidelity rather than rich physical interactions; your robot can navigate around a chair, but cannot simulate pushing it yet. Even so, solving spatial consistency at 131MB is an exceptionally sharp move.