Meta's ShapeR turns messy videos into editable 3D objects

Meta's ShapeR takes a casual video or multi-view photo sequence and reconstructs it into individual, metrically accurate 3D meshes.

Traditional 3D scene captures like NeRF or Gaussian Splatting usually bake a whole room into a single, uneditable blob. You cannot easily pull a chair out to use in a game engine or slide a coffee cup across the table. ShapeR exists to fix that exact editability headache.

How it turns video into separate objects

Instead of reconstructing the room as one giant mass, ShapeR works object-centrically. It relies on standard multimodal capture data rather than custom sensor setups.

First, off-the-shelf SLAM tracks the camera and generates sparse 3D point clouds, while 3D instance detection spots the objects in the room. For each detected object, ShapeR pulls out its multi-view crops, 2D projections, SLAM points, and VLM-generated text descriptions.

Next, a rectified flow transformer takes these multimodal cues as conditions to denoise a latent representation called VecSet, finally generating clean 3D meshes. Because it anchors itself to multi-view camera poses and SLAM points, the objects are metrically accurate in real-world scale, layout, and aspect ratio.

Single-image tools like SAM 3D often struggle with warped proportions in cluttered rooms. ShapeR solves this geometry problem automatically, without requiring human clicks or interactive bounding boxes.

Training, specs, and open weights

What makes the architecture stand out is that ShapeR was trained purely on synthetic data from the Aria Synthetic Environment. The team used heavy augmentations and curriculum training, intentionally throwing in clutter, occlusions, and noisy SLAM points to simulate the real world.

As a result, it generalizes out of the box. It handles unconstrained real-world environments like ScanNet++ without extra fine-tuning. Paired with depth-metric predictors like MapAnything, it can even estimate metric shapes from single images without retraining.

Meta also made ShapeR combinable with SAM 3D: ShapeR locks down the scale and layout, while SAM 3D injects detailed real-world textures and priors.

Everything is available on GitHub and the project page. Meta has released the codebase, pre-trained model weights, and an in-the-wild benchmark dataset featuring 178 annotated objects across 7 messy real-world scenes.

Why it matters

If you build 3D environments, game assets, or train embodied AI agents, ShapeR is well worth testing. It lets you walk around a room with a smartphone and walk away with clean, individually movable 3D assets that actually match real-world dimensions.

It cleanly bridges classical geometric SLAM with modern generative flow models. While raw video captures with heavy motion blur will always pose a challenge, turning passive spatial video into modular, metric building blocks is a huge practical step forward for digital twins and spatial computing.