ProxyPose tracks 3D object poses from regular 2D video

Tracking the exact 3D position and rotation of an object from a flat, single-camera video has always been a headache. Throw in a transparent wine glass, a shiny metallic surface, or a hand completely blocking the view, and classical computer vision trackers usually throw in the towel.

Researchers from the Toronto Computational Imaging Group open-sourced ProxyPose, accepted at NeurIPS 2026. The pitch sounds almost too simple: feed in a regular 2D video, click a single query pixel on an object, and get a full 6-DoF 3D pose trajectory back.

Swapping complex objects for a colored cube

Instead of struggling to track messy, reflective real-world surfaces directly, ProxyPose pulls off a clever sleight of hand using video-to-video translation.

When you select a point, the system employs an underlying video diffusion model to synthesize a "proxy video." In this alternate version, your selected object region is replaced by a clean, multi-colored cube that moves with the exact same local rigid-body motion as the original target.

Because the cube has crisp edges, known geometry, and distinct colors, the hard perceptual problem evaporates. The system simply runs a standard Perspective-n-Point (PnP) geometric solver on the synthetic cube's contours to extract the 6-DoF trajectory frame by frame.

This proxy trick effortlessly cuts through scenarios where traditional tools fail, including transparent items, mirror reflections, and heavy occlusions. On standard rigid tracking benchmarks like HO3D and YCBInEOAT, a single query point in ProxyPose beats state-of-the-art baselines. Remarkably, it can also track non-rigid surfaces like human faces, assist camera pose estimation when COLMAP chokes on textureless walls, and work zero-shot on unconventional sensor feeds like event cameras and single-photon SPAD arrays.

The power and baggage of Wan 2.1

Under the hood, ProxyPose runs on the Wan 2.1 video foundation model. While the full model sits around 30GB, the custom LoRA adapter driving the colored cube generation is just 600MB, with quantized versions available for consumer-tier GPUs.

However, leaning on a large video model comes with clear trade-offs.

The authors note that fast-moving scenes, like high-speed marble racing, can overwhelm the video model's variational autoencoder (VAE), creating motion blur that ruins edge detection. Textureless reflective surfaces under complex movement can still suffer from pose drift, and naturally, tracking breaks down completely on ill-defined physical states like fluid surfaces.

Why it matters

We think ProxyPose is a brilliant example of using generative video models not for entertaining clips, but as intermediate geometric translators for hard spatial perception.

Foundation models hold vast, implicit physical knowledge about how objects move through 3D space. ProxyPose exposes that latent understanding by translating ambiguous visual noise into a mathematically tractable toy block. For robotic grasping, AR effects, and spatial computing, this unlocks a low-cost bridge from raw mono video to spatial metrics.

It is certainly not real-time yet, and video VAE bottlenecks mean high-speed robotics will have to wait. But by proving that a 600MB LoRA can turn an off-the-shelf video generator into a world-class 6-DoF pose estimator, the researchers have fundamentally expanded what video models are useful for.