Ai2 open-sources WildDet3D for instant 3D object detection from a single photo

Most vision AI today can instantly point out a chair, a mug, or a laptop in a photo. But ask it how far away that chair is, exactly how wide it measures in meters, or how it is tilted, and the system usually goes blind.

That limitation is the core hurdle facing spatial intelligence. Whether you are building warehouse robots sorting packages, self-driving platforms navigating streets, or AR overlays guiding pedestrians, machines need to understand physical dimensions. Historically, getting true 3D bounding boxes from visual feeds required multi-camera setups, costly LiDAR rigs, or models confined to narrow driving benchmarks.

Ai2 open-sourced WildDet3D, a monocular 3D detection system designed to solve this across diverse environments. Give it a single standard photo or camera feed, and it predicts complete 3D bounding boxes with real metric dimensions, orientation, and spatial distance.

Text, clicks, and zero-shot tracking

What makes WildDet3D immediately practical is how flexibly you can tell it what to look for.

Users can pass natural language prompts like monitor or paper, and the model scans the frame to place accurate 3D boxes around every match. It works across crowded scenes with dozens of items, from furniture in an apartment to animals in the wild.

If you prefer manual control, you can simply tap an object on a screen or draw a standard 2D bounding box. WildDet3D takes that simple 2D input and immediately lifts it into a full 3D extent.

Because it can ingest 2D boxes from any upstream detector frame by frame, it also enables zero-shot 3D tracking in video. Even though it was never trained on explicit tracking sequences, it can maintain persistent spatial awareness of objects as the camera moves around.

Under the hood: SAM3 meets DINOv2

To keep performance fast without sacrificing geometry, Ai2 split the workload into parallel tracks.

First, a 2D detector built on top of the SAM3 vision backbone handles the multimodal prompts. Running alongside it is a geometry branch powered by a frozen DINOv2 encoder and a trainable depth decoder, estimating per-pixel depth. A lightweight 3D detection head then fuses the two feature streams using cross-attention to predict the final metric bounding box.

When extra sensor data like sparse LiDAR or time-of-flight depth is available, the model folds it in seamlessly to sharpen accuracy. Ai2 even released an iOS demo app running real-time AR bounding boxes directly on an iPhone with live camera and LiDAR.

Backing the release is WildDet3D-Data, an open dataset of over 1 million images featuring 3.7 million verified 3D annotations across more than 13,000 object categories, including over 100,000 human-annotated examples. On April 21, the team followed up by releasing the complete training code and data prep pipelines.

Why it matters

Giving AI spatial intuition has often felt like an expensive, closed-door race dominated by autonomous driving labs and hardware giants.

We think Ai2 took a remarkably smart architectural shortcut here. Instead of forcing an oversized vision-language model to hallucinate geometry, they built a modular spatial reasoning layer that can attach to existing pipelines. You can pair it with multimodal models like Molmo 2 to answer complex queries, or slot it into a drone camera feed for real-time localization.

For robotics engineers and independent spatial computing builders, this eliminates one of the biggest bottlenecks: high-quality 3D data. Handing over 3.7 million verified annotations and the training scripts lets anyone adapt metric 3D perception to custom hardware.

The caveat is that while the lightweight demo proves on-device AR is feasible, running the uncompromised full model continuously at high frame rates still calls for server hardware or further compression. Even so, by open-sourcing the entire stack, Ai2 just leveled the playing field for open-world spatial intelligence.