Google DeepMind unveils D4RT for real-time 4D dynamic tracking

If you asked a computer vision engineer a few years ago to reconstruct a moving scene in 3D from standard video, they would have handed you a pipeline of five fragile models and told you to rent a render farm.

You had to estimate optical flow, guess monocular depth, track camera poses, and pray the global optimization didn't collapse into a blurry mess of ghost artifacts.

Google DeepMind unveiled D4RT, short for Dynamic 4D Reconstruction and Tracking. It throws away the multi-stage pipeline entirely.

Instead of patching together separate tools, D4RT relies on a unified Transformer framework that ingests standard 2D video and reconstructs dynamic 3D space across time, delivering up to 300x higher tracking throughput than prior state-of-the-art systems.

Turning 4D perception into a search engine

Traditional 3D reconstruction breaks down the second something moves. A swan gliding across water or a flower swaying in the breeze usually leaves a trail of duplicate limbs and noise, because the math assumes a frozen world.

D4RT treats the problem like an indexed database. A 1-billion-parameter ViT-g encoder first processes the entire video clip into a global spatio-temporal scene representation.

Once this global memory is built, any 4D task becomes a single, lightweight query: tell me the 3D position of this specific pixel at time T, as viewed from camera pose C.

If you want dense depth, you query the current frame. If you want full-pixel tracking, you ask where that pixel went in every subsequent frame, even if it gets temporarily occluded behind an object. DeepMind even feeds a 9x9 RGB patch into the query to stop the model from confusing identical textures.

Because every query runs independently, the entire workload scales across hardware in parallel. A complex serial geometry problem suddenly becomes a fast lookup job.

Breaking down the 300x speedup

DeepMind claims D4RT can process a one-minute video clip in just five seconds.

It is worth noting where that speedup actually lives. The 300x claim refers to trajectory throughput. Running at 24 frames per second, previous leading models like SpatialTrackerV2 hit a wall around 84 simultaneous 3D tracks. D4RT handles 1,570 tracks at the same frame rate, which is over 300 times faster than older baselines like DELTA.

Instead of picking a single foreground subject to follow, the system can track thousands of points at once, from background traffic to swirling leaves.

The catch is the training bill. Pre-training this 1-billion-parameter monster required 64 TPU chips running non-stop for two full days. It is not an indie weekend project; it is big-lab infrastructure.

Why it matters

We see D4RT as a critical stepping stone for embodied AI and next-generation mixed reality.

Robots have long struggled to move safely through unpredictable human spaces. Static mapping is easy, but anticipating where a running pet or an approaching pedestrian will be half a second from now requires dense dynamic intuition. D4RT turns that temporal bottleneck into real-time numbers.

For consumer tech, this approach could turn casual smartphone videos into fully navigable 4D scenes, letting users freeze time, swing the camera around, or cleanly erase background distractions.

DeepMind made a clever bet here. By replacing brittle pipelines with a unified query interface, they unlocked massive parallel throughput. The main hurdle now is compute: slimming down that massive encoder so it can actually run locally inside an AR headset or a lightweight robot dog.