ByteDance's Lucida turns messy rooms into fully editable 3D scenes

If you've ever 3D-scanned a room, you know the frustration: you get back a gorgeous visual mesh, but it's basically a single frozen rock. The mug is fused to the desk, the desk is glued to the floor, and nothing can be individually moved or manipulated.

ByteDance Seed, alongside Peking University and Zhejiang University, introduced Lucida. It is a composable 3D scene modeling pipeline designed to turn messy, real-world indoor video captures into interactive digital twins where every object is an independent, editable asset ready for simulation.

Parse, generate, and place

Standard pipelines typically break real-to-sim reconstruction into three phases: parse the objects, build individual models, and place them back. The problem is they expect pristine inputs from the start, which a cluttered real-world room almost never provides.

Lucida keeps the three-stage sequence but flips the expectations. It expects noisy inputs upfront and saves high-precision alignment for the very end.

First, it parses keyframes across views to identify instances and construct a scene graph. Second, it synthesizes an unoccluded view for each object even if it was partially hidden, lifting it into a complete 3D mesh. Finally, instead of guessing poses in one shot, Lucida introduces GizmoAct, a vision-language model policy that acts like an artist in a 3D editor. GizmoAct looks at multi-angle rendered feedback, adjusts the object's gizmo frame by frame in a closed loop, and stops when visual alignment is reached.

In benchmarks, this approach delivered huge leaps. On R2S-Scene, Lucida boosted 3D object detection mAP by 69% over Boxer. On CA-1M, object pose accuracy ADD-SB@0.05 surged from 57.8% to 83.4%, while scene F-Score jumped to 0.924, soundly beating SAM3D's 0.794.

Why it matters

Embodied AI and robotics have a massive simulation bottleneck. Training generalist robots requires thousands of realistic interactive rooms, but hand-crafting CAD environments takes forever and raw Gaussian splats or photogrammetry meshes can't be interacted with.

Lucida offers a practical blueprint for turning smartphone room walkthroughs into functional training grounds. You capture a messy living room, and the system hands back individual cups, chairs, and tables ready to be picked up or rearranged.

We think using a VLM as an iterative 3D editor is a very clever architectural shift. Instead of hoping an end-to-end model guesses the 6D pose right on the first pass, letting an agent visually inspect and nudge the gizmo mirrors how human modelers actually work. For now, it remains a research paper without public code, but if it runs reliably at scale, robot simulation pipelines just found their fast forward button.