Tencent's VerseCrafter lets you choreograph AI video trajectories inside Blender

Anyone who has tried coaxing an AI video model into a specific camera move knows the pain. You type "slow pan right while the car accelerates," and the model gives you a disorienting warp where the car melts into the pavement.

Tencent researchers just dropped VerseCrafter to fix that choreographic disconnect. Instead of relying on lucky text prompts, it lets you bring a single 2D image into Blender, set up actual camera dollies, move foreground objects along custom paths, and have the model generate physically coherent video along those rails.

Bringing 2D frames onto a 3D stage

The trick behind VerseCrafter is treating video generation like virtual production rather than pure pixel hallucination.

Feed it a single image, and the system runs MoGe for depth estimation alongside SAM 2 for object segmentation. Together, they lift that static picture into a 3D point cloud scaffold. Foreground subjects are represented as 3D Gaussian trajectories, while the background forms a spatial geometry rig.

The real workflow win is Blender integration. Creators get a dedicated plugin where they can intuitively script camera paths and drag 3D Gaussian objects across the viewport. You can freeze the camera and move three different props, or move both camera and actors at once. These 3D paths are rendered into per-frame 4D geometric control maps.

Rather than training a video foundation model from scratch, VerseCrafter mounts onto a frozen Wan2.1 diffusion backbone. Keeping the base model untouched preserves visual realism, while a lightweight GeoAdapter injects the geometric signals into selected diffusion layers via zero-initialized residual connections. Reports note that compared to view-synthesis setups like YuMe or Uni3D, VerseCrafter produces notably sharper temporal stability.

To power this, the team built VerseControl4D, an annotated dataset of 35,000 training clips and 1,000 validation clips pulled from Sekai-Real-HQ (26%) and SpatialVID-HQ (74%). Roughly 20% of the clips are purely static environments, designed explicitly to evaluate camera-only exploration.

Where the rails still slip

VerseCrafter is impressive, but the authors are candid about its geometric boundaries.

Right now, object rotation is handled via ellipsoid principal axes rather than full 6D pose estimation. While this works smoothly for rigid items like vehicles or boxes, single ellipsoids struggle with human bodies. Because a standing person has an almost circular horizontal cross-section, the system can get confused about facing direction, occasionally mismatching where someone is actually looking.

Background physics is another friction point. Because the background mesh is anchored to a static point cloud from frame one, it does not account for dynamic deformations over time. Fine, texture-heavy dynamics like rushing waterfalls or breaking waves lose structural enforcement during the 3D-to-2D projection, leaving the synthesis noticeably looser than the rigid camera paths.

Why it matters

Generative video is rapidly outgrowing the "gacha slot machine" phase. For animators, storyboard artists, and game designers, raw video fidelity is useless without deterministic spatial control. VerseCrafter provides an elegant bridge by connecting modern diffusion models directly to standard digital content creation (DCC) software like Blender.

Freezing Wan2.1 and outsourcing spatial guidance to a lightweight adapter is also a clever architectural blueprint. It proves teams do not need massive compute budgets to retrain 14B parameter models just to enforce geometric consistency. By meeting visual creators where they already work—in 3D viewports—VerseCrafter points toward a pipeline where AI handles realistic rendering while the human stays firmly in charge of the camera crane.