UniSH reconstructs 3D scenes and moving humans in a single pass

UniSH is an open-source framework that reconstructs full 3D scenes and moving human bodies directly from a single standard video. Instead of juggling multiple disconnected tools, it processes both the environment and the actor together in a single feed-forward pass.

What it does

Feed it an ordinary in-the-wild video, and UniSH simultaneously predicts four separate outputs: the 3D scene point cloud, exact camera motion, metric-scale SMPL human bodies over time, and a dense human point cloud.

Because everything is predicted jointly in the same metric coordinate system, the human body stays properly grounded on the floor without floating or clipping through geometry.

Even when the camera is shaky and the human subjects perform complex motions, the framework tracks movements consistently, allowing you to freely render and inspect the dynamic scene from any viewpoint.

How it works

Historically, 3D reconstruction has been fractured. Scene reconstruction tools ignored moving people, while human mesh recovery models ignored the surrounding room. Training unified models was also held back by the scarcity of annotated real-world video datasets, leaving models trained on synthetic data struggling on real footage.

UniSH bypasses slow per-video optimization pipelines entirely by relying on a feed-forward neural network.

To bridge the gap between synthetic data and real-world videos, the authors introduced two key components.

First, a robust distillation strategy borrows high-frequency detail from an expert depth model, sharpening the geometric resolution of the human surface.

Second, a two-stage supervision pipeline trains the model first on coarse localization using synthetic datasets, and then fine-tunes on unannotated real-world videos. It does this by directly aligning the geometric correspondence between the SMPL mesh and the human point clouds, solving the sim-to-real gap without requiring manual 3D annotations.

How to get started

The code is available on GitHub alongside an official project page. The authors have included offline installation instructions and execution steps for setting up local dependencies and running forward inference on custom video clips.

Why it matters

For creators and developers working in game development, spatial computing, and robotics, UniSH eliminates the headache of manually stitching camera tracking, motion capture, and background geometry together.

Replacing iterative optimization with a feed-forward pass makes 3D generation dramatically faster. If you need to turn casual video footage featuring moving people into functional 3D assets, UniSH is definitely worth testing in your local workflow.