ByteDance's UI-TARS controls your desktop and mobile apps
ByteDance just expanded its open-source UI-TARS agent series, rolling out UI-TARS-2 with upgrades across GUI workflows, coding, and tool use. Built on native vision-language models with reinforcement learning reasoning, it translates screenshots into system clicks, drags, and keystrokes. Why it matters: It turns complex computer and mobile tasks into reliable, hands-free automation.
UI-TARS is an open-source multimodal agent from ByteDance that navigates computers and smartphones just like a human, looking at raw screens to click buttons, type text, and drag windows.
ByteDance announced UI-TARS-2, an all-in-one upgrade over UI-TARS-1.5 that sharpens the model's skills across GUI control, gaming, coding, and tool use.
What it can do
UI-TARS provides specialized prompt templates for different platforms.
On Windows, macOS, and Linux desktops, it executes single and double clicks, right clicks, drag-and-drop actions, keyboard shortcuts, typing, and page scrolling. That makes it handy for browser research, office software workflows, and desktop file management.
On Android devices and emulators, it supports mobile gestures like long presses, home and back navigation, and opening apps.
It also handles interactive environments surprisingly well. In official tests, UI-TARS swept perfect scores across several puzzle titles on the Poki gaming platform and outperformed previous reinforcement learning baselines in Minecraft tasks like gathering resources and combating mobs.
Under the hood and benchmarks
UI-TARS is built on vision-language backbones and maps out interface elements using absolute coordinates. By integrating reinforcement learning, the system forces the model to deliberate on its thoughts before triggering an action.
That pause for reasoning pays off in benchmark scores. On the 100-step OSWorld computer-use benchmark, UI-TARS-1.5 scored 42.5, topping OpenAI CUA at 36.4 and Claude 3.7 at 28.
In the ScreenSpotPro grounding benchmark, it hit 61.6, running well ahead of rival models.
Still, the creators note clear constraints. It demands significant compute over extended gameplay, occasionally hallucinates actions in unfamiliar interfaces, and poses security concerns given its knack for solving authentication challenges like CAPTCHAs.
How to run it
The project is available on GitHub under bytedance/UI-TARS. You can install the core Python library via pip install ui-tars, which parses model responses directly into executable pyautogui scripts.
If you prefer a ready-to-run personal app, ByteDance provides the UI-TARS-desktop repository. For browser testing and web automation, it integrates with Midscene.js.
Pretrained weights including UI-TARS-1.5-7B are hosted on Hugging Face, while top-tier model checkpoints can be requested for collaborative research.
Why it matters
If you want to automate repetitive tasks across disparate desktop apps without fiddling with fragile DOM selectors or custom APIs, UI-TARS is one of the most capable open-source options available today.
We think open-sourcing a native vision-action stack puts genuine pressure on proprietary computer-use features from Anthropic and OpenAI. By rolling code execution and tool integration into UI-TARS-2, ByteDance is making locally run, hands-free personal screen assistants feel much closer to a daily reality.
Sources (1)
- github.com(opens in a new tab)Originalgithub.com