OVOW turns monocular video into physics-ready 4D scenes

OVOW (One Video, One World) is a training-free 4D scene reconstruction system that turns a regular monocular video or a single image into physically interactive 3D assets.

Instead of outputting a fused blob of geometry or an uneditable neural field, OVOW decomposes a scene into separate, watertight object meshes. Each object gets its own 6-DoF pose trajectory and deformation animation, assembled into an animated GLB and URDF file ready to drop straight into physics engines.

How it turns video into a physical scene

OVOW does not train a massive end-to-end network from scratch. Instead, it wires together several state-of-the-art foundation models into a modular four-stage pipeline.

First, scene understanding and segmentation. It uses Qwen3-VL to detect and classify instances as static, rigid, or deformable, while Meta's SAM 3 generates dense per-frame object masks.

Second, asset reconstruction. For rigid and static objects, FLUX.2 inpaints occluded areas before Hi3DGen reconstructs watertight 3D meshes. For deformable instances, Motion324 paired with Hunyuan outputs topology-consistent dynamic mesh sequences, with VGGT recovering metric scale.

Finally, pose optimization and assembly. An iterative loop using RoMa v2 and FoundationPose recovers orientation, metric scale, and per-frame motion trajectories, decoupling global movement from local vertex deformation. The system estimates the ground plane and contact relationships, ensuring assets rest, collide, and topple naturally under gravity.

Hardware specs and license details

The pipeline is compute-heavy. It requires Linux, CUDA 12.1+, roughly 120GB of disk space, and recommends an NVIDIA GPU with at least 40GB of VRAM.

Processing a 108-frame clip takes about 19 to 22 minutes end-to-end. To speed up multi-video workflows, heavy backends like SAM 3, FoundationPose, and RoMa v2 can run as resident services, loading weights once for an entire batch.

The project is available on GitHub. While OVOW's own orchestration code carries an MIT license, it bundles third-party dependencies with restrictive terms, including NVIDIA FoundationPose, SpaTrackerV2, and Motion324. As a result, the release is restricted to non-commercial research use only.

Why it matters

For researchers in embodied AI, robotics, and game physics, OVOW is an impressive demonstration of how far modular foundation models can go. Instead of training an opaque black-box world model, chaining vision, inpainting, and pose estimators produces structured, physics-ready assets directly from real-world footage.

That said, the 40GB VRAM requirement, complex multi-environment setup, and non-commercial licensing mean you won't be shipping this inside a commercial production tool tomorrow. For now, it serves as a powerful blueprint for automated simulation asset generation.