Nvidia drops Describe Anything 3B for pinpoint image and video captions

Nvidia just open-sourced Describe Anything 3B (DAM-3B), a 3-billion-parameter multimodal model built specifically to generate rich, localized captions for specific regions in images and video.

While most general vision-language models do great at summarizing entire scenes, they usually fall flat when asked to inspect a tiny detail in the background or describe a single moving object in a video clip. DAM-3B exists to solve that exact blind spot.

What it does

DAM-3B accepts flexible region inputs. You can highlight a target by clicking a single point, drawing a bounding box, sketching a scribble, or feeding it an exact segmentation mask.

Once a region is flagged, the model generates a grounded, descriptive caption while keeping the full image context in mind. Alongside the base model, Nvidia dropped DAM-3B-Video, which tracks frame-wise masks over time so you can inspect objects in motion even when occlusions happen.

Both weights and research papers are publicly accessible on Hugging Face and the project page.

How it works

Zooming into an object usually strips away surrounding scene context. Nvidia bypassed this trade-off using a dual-view technique called a focal prompt.

The system pairs the full image with a high-resolution crop of the target region. A localized vision backbone then fuses these views using gated cross-attention before handing the data over to the language model. Crucially, it blends these details without blowing up token length, keeping inference fast.

For video, DAM-3B-Video encodes region masks across individual frames and aggregates them temporally to handle rapid movement and visual interruptions.

To train it without relying on scarce manual annotations, Nvidia built a semi-supervised pipeline named DLC-SDP. By pairing existing segmentation datasets with unlabeled web images, they curated 1.5 million localized training examples and refined the text using self-training.

Benchmark results

Nvidia also introduced DLC-Bench, an evaluation standard that measures attribute-level precision rather than relying on brittle word-matching against reference captions.

On DLC-Bench, the 3B model hit an average accuracy of 67.3%, beating out GPT-4o and specialized setups like VideoRefer in both granularity and correctness.

It also took top scores across keyword and phrase-level benchmarks like LVIS, PACO, and Flickr30k Entities, as well as complex multi-sentence datasets including Ref-L4 and HC-STVG.

Why it matters

We think DAM-3B shows how specialized small models can outmaneuver frontier giants. Pushing generalized vision models to hundreds of billions of parameters is great, but robotics, accessibility tech, and video editing tools don't always need open-ended chats; they need to know exactly what is inside a 20-pixel box.

By packaging this into a 3B footprint, Nvidia gives developers a model light enough to run on local workstations or edge servers without paying constant API tolls to frontier model providers.

If you work on video search, spatial computing, or robot perception, this is worth testing immediately. Just remember that it is built for pinpoint inspection, not high-level abstract reasoning, so you will likely want to pair it with a broader reasoning model in production.