Anthropic puts AI in the drone pilot seat

Could a frontier AI model take a store-bought drone, look at a single reference photo, and hunt down a specific person through an office maze?

Anthropic and Andon Labs just put that question to the test with Project Pilot, introducing a new benchmark called Drone-Bench. They pitted 15 models from three developers—including GPT-4o, Gemini 3.1 Pro, GPT-5.6 Sol, and Claude Fable 5—against the physical world.

Instead of a black-box test, the benchmark breaks aerial surveillance down into five distinct sub-tasks: reconstructing a 3D obstacle map from video, localizing the drone within that map, navigating between rooms, detecting the target face, and following the person while maintaining camera framing.

Claude Fable 5 nearly pulled it off

Across the board, newer models showed dramatic improvements, but Claude Fable 5 took the crown. It beat the human-assisted engineering baseline on four out of five tasks, shining brightest at visual detection and dynamic target following.

When it came to flying an actual drone through real office hallways, however, Fable stumbled. The reason was 3D reconstruction. The model failed to accurately turn video feeds into a clean spatial map, and those geometric errors cascaded directly into localization and path planning.

Yet the way Fable tackled the challenge caught researchers off guard. The model demonstrated genuine tool-making intuition, iterating locally before executing commands.

In one instance, Fable analyzed simulation footage, traced the grout lines on the office floor to locate the vanishing point, and calculated the drone camera tilt within four degrees of ground truth. In another run, it coded a 2D top-down mini-simulator to debug its flight logic before submitting code to the real aircraft.

Consistency remains a glaring hurdle: over 10 simulation runs, Fable 5 averaged baseline performance on only three of the five tasks.

Why it matters

Deconstructing physical robotics into discrete software challenges reveals exactly where frontier models stand today. They are phenomenal at perceptual tasks like tracking and spotting faces, but they still struggle with rigid geometric reasoning and 3D spatial reconstruction.

We think this evaluation proves two things. First, fully autonomous aerial tracking driven solely by off-the-shelf multimodal models isn't quite here today—the missing reconstruction link keeps the physical world safe for now. Second, that gap is closing uncomfortably fast.

By open-sourcing Drone-Bench, Anthropic is highlighting the dual-use dilemma. The hardware is cheap, consumer-grade, and widely available. As soon as models solve the mapping bottleneck, autonomous drone tracking will shift from specialized military tech to commodity code anyone can run.