AutoRT, SARA-RT, and RT-Trajectory boost robot speed and skill

Teaching a robot to wipe down a kitchen counter is notoriously harder than training an AI to pass the bar exam. The real world doesn't have an internet-scale dataset of physical interactions, robots can be painfully slow to process sensor data, and the moment you change the lighting or move a mug, their performance falls apart.

Google DeepMind just dropped a trio of research breakthroughs targeting this exact bottleneck: AutoRT, SARA-RT, and RT-Trajectory. Together, they tackle data collection, processing speed, and physical generalization.

LLMs as robotic choreographers

Getting robots to understand practical human goals requires massive amounts of diverse physical data. DeepMind's answer is AutoRT, a system that essentially turns large models into autonomous job dispatchers for robot fleets.

Here is how it works: a robot uses a vision-language model to scan its immediate environment and spot objects. Then, a large language model brainstorms sensible tasks it could perform on the spot—like placing a snack onto the countertop—and picks the best one for the robot to execute.

Over a seven-month trial in actual office buildings, AutoRT orchestrated as many as 20 robots at the same time and managed 52 unique robots overall. In total, it racked up 77,000 real-world trials across 6,650 distinct tasks.

To keep these machines from wrecking the office, DeepMind introduced a Robot Constitution inspired by Isaac Asimov’s laws. The system explicitly forbids actions involving humans, animals, sharp objects, or electrical appliances. Still, DeepMind didn't trust software prompts alone: robots automatically freeze if joint force exceeds strict thresholds, and human monitors kept watch with physical kill switches.

Leaner Transformers and visual sketchpaths

Collecting data is useless if robots react too slowly. Robotics Transformers rely on attention mechanisms with quadratic complexity, meaning double the camera resolution requires four times the computing power.

DeepMind introduced SARA-RT, which uses an up-training technique to convert that quadratic complexity into linear complexity without expensive pre-training. Applied to a multi-billion parameter RT-2 model, SARA-RT improved accuracy by 10.6% while cutting latency by 14%. When applied to Point Cloud Transformers processing 3D depth cameras, it doubled execution speed.

Meanwhile, the third piece of the puzzle, RT-Trajectory, tackles generalization. Instead of trying to connect vague instructions like wipe the table directly to raw motor torque commands, it overlays a 2D visual trajectory of the robotic gripper onto training videos.

This simple visual trail teaches the robot how to move in physical space. It can even take hand-drawn sketches or videos of humans doing the task. When tested on 41 completely unseen tasks, RT-Trajectory scored a 63% success rate, compared to just 29% for baseline RT-2.

Why it matters

We think this release signals a major shift in how robotic systems will be built moving forward.

For years, robotics has been choked by the data problem. Teams spent endless hours manually teleoperating arms or relying on synthetic simulation data that failed to translate to messy real rooms. AutoRT proves that foundation models can successfully automate real-world physical training at scale, cutting data collection overhead dramatically.

SARA-RT is equally practical. Because it works as an efficient fine-tuning recipe without requiring full model retraining, it lets roboticists pack far more intelligent visual reasoning onto lower-power edge hardware that actually fits inside a mobile robot.

The catch, of course, is safety. DeepMind's own setup still required human monitors holding physical deactivation switches. Until we can trust autonomous robotic constitution prompts without a human finger hovering over an emergency kill button, truly unmonitored deployment in our homes remains out of reach.