4DAnyone turns casual phone videos into interactive 4D avatars

4DAnyone is a new open-source framework that turns any casual phone video of a moving person into an interactive, photorealistic 4D Gaussian Splatting avatar.

Drop in a single monocular video clip, and it spits out a dynamic 3D asset that you can rotate around, pause, and inspect from any angle in real time. It handles various body shapes and loose clothing out of the box.

How it pulls off novel-view consistency

Reconstructing a moving person in 4D usually falls apart because single-camera diffusion models struggle to keep dozens of novel viewpoints consistent. If you have ever tried tools like ReCam Master or Trajectory Crafter, you know the drill: generate more than a handful of angles, and the subject’s face morphs or structural drift sets in.

The 4DAnyone team tackled this bounded-attention bottleneck with two clever mechanics under the hood.

First, the framework extracts a 3D skeleton from your input clip to anchor the spatial movement.

Then, instead of choking the diffusion model with infinite context, it uses Reference Context Packing (RCP) to squash prior views into an O(1) mixed-resolution context. Alongside that, Target Context Routing (TCR) shuffles view groupings across denoising steps, sharing high-level structure early on before locking down fine details. Once all the synchronized viewpoints are rendered, it lifts the entire sequence straight into 4D Gaussian Splatting.

Specs and what is in the box

The researchers trained the system across a hybrid mix of real-world captures, light-stage data, and a custom synthetic game-engine dataset dubbed MVGameHuman.

In benchmarks like DNA-Rendering and DyMVHumans, 4DAnyone visibly outperforms existing baseline models in multi-view fidelity and temporal stability.

The team has fully open-sourced the pipeline. You get the complete codebase, local setup instructions, and the trained model weights, which clock in at around 12GB.

Running this locally will demand a decently beefy GPU with plenty of VRAM, but the inputs can be as casual as an uncalibrated video recorded on your smartphone.

Why it matters

For indie game devs, VFX animators, and digital avatar creators, this is worth testing immediately. Dynamic 4D volumetric capture used to require a cage of fifty synced cameras in a dedicated studio. Getting comparable multi-view consistency from a single video takes a massive bite out of production budgets.

That said, we think single-camera 4D reconstruction is still best treated as an early-stage asset generator rather than a final production pipeline. Because unseen angles rely heavily on diffusion guesswork, wild acrobatics or messy occlusions can still cause slight geometric artifacts. Even so, 4DAnyone sets a genuinely impressive high-water mark for open-source 4D avatars.