Berkeley and Tsinghua open-source uni-robot-agent to give open LLMs physical hands

uni-robot-agent is an open-source framework that wraps open-weight LLMs in a standardized harness, allowing them to directly control robotic arms in both simulators and the physical world.

Until now, running a zero-shot robot-use agent usually meant shelling out money for proprietary APIs like GPT-6 Astra. Because physical tasks take dozens of iterative steps, API bills stack up fast. Worse, closed weights mean you cannot take the interaction logs and train the model locally.

How it plugs models into physical hardware

The team from Tsinghua University and UC Berkeley tackled the plumbing instead of fine-tuning a custom backbone. They built a unified, modular harness that connects open models, simulators, and real robots through identical tool interfaces.

The system splits the workflow into swappable components: agent runtime, context management, perception backends, and motion tools. More importantly, it bans common simulation cheats. The model gets no ground-truth depth, no CAD models, and no real-time success flags. It only sees fixed-camera RGB streams, standard camera calibration parameters, robot URDF descriptions, and natural language prompts.

To keep hardware from dragging down inference, it also introduces asynchronous pipelining. Physical motions execute in the background while the model already plans the next step, avoiding awkward serial pauses.

Local Qwen tops the Astra baseline

With proper geometric scaffolding in place, open models punch well above their weight.

On the 42-task RoboDojo benchmark, a locally hosted Qwen3.8-Flash-Next scored 31.39, edging past the baseline of GPT-6 Astra running RoboProbe at 28.97. DeepSeek-V4.1-Flash also notched 27.11 points. The biggest gap is operating cost: running Qwen costs roughly 1 cent per task, about 1/20th of the proprietary alternative.

In real-world testing across 19 tabletop tasks on a bimanual setup, Qwen hit 53.50 points without ever training on physical demonstration data. That beat both zero-shot Dream-1 and the domain-fine-tuned π0.5 (47.75 points). The team has also ported the harness to Franka arms paired with Sharpa dexterous hands for pressing buttons and grasping plush toys.

The entire codebase, docs, and evaluation suite are up on GitHub under Open-RUA/uni-robot-agent.

Why it matters

If you build robotics software or embodied AI setups, this is worth cloning today. It pulls embodied research away from expensive cloud APIs and gives you a reproducible local baseline where model weights, trajectories, and tool definitions are completely under your control.

Keep in mind its boundaries, though. The ablation tests show that if you remove camera calibration matrices, Qwen's performance drops sharply because open models still struggle with pure visual 3D guessing. Complex long-horizon planning and sub-millimeter fine manipulation also remain unsolved bottlenecks. Still, as an open-source bridge between off-the-shelf LLMs and physical actuators, it is one of the cleanest architectures built so far.