ShowUI-π teaches computer-use agents how to actually drag and drop

ShowUI-π is a lightweight 450M parameter vision-language-action model designed to fix one of the most frustrating flaws in GUI agents: dragging, drawing, and handling continuous mouse movements.

Instead of treating screen interactions as isolated clicks, it models actions as continuous trajectories, allowing agents to move cursors just like human hands.

What it can do

Most modern computer-use agents can identify buttons and type text, but they fall apart whenever a task requires holding down the mouse button and dragging smoothly across the screen.

ShowUI-π specifically targets this gap. It can handwrite text or draw sketches in digital canvases, drag media clips across timelines in Adobe Premiere Pro, resize shapes in PowerPoint, and organize files across folders.

It is also surprisingly good at interactive captchas. Whether dealing with sliding puzzles, dial rotations, or trajectory-based verification, it continuously observes visual changes and adjusts cursor movement on the fly.

NUS researchers evaluated it against ScreenDrag, a benchmark featuring 505 manipulation tasks across five domains and 20,000 dense trajectory demonstrations.

ShowUI-π hit a 26.98% overall online success rate with the lowest trajectory error, beating open-source heavyweights like OpenCUA-7B (21.98%) and proprietary giants like Gemini-2.5-CUA (22.18%).

Its lead is especially striking in continuous tasks. It scored 55.91% on captchas compared to Gemini's 3.96%, and 34.32% on handwriting where most existing agents scored zero. In Premiere Pro, it reached 8.64% while most competitors failed completely.

How it works

Traditional GUI agents predict discrete click coordinates using language tokens. That setup works for single clicks, but it breaks down on long drags and cannot support real-time closed-loop adjustments.

ShowUI-π takes a different approach by treating discrete clicks and continuous drags as sequences of cursor waypoints paired with mouse states.

Using flow matching, the team trained a lightweight action expert that takes streaming visual observations and continuously outputs incremental cursor deltas.

Because it generates fluid paths rather than jumping between discrete tokens, cursor movements stay stable, continuous, and responsive to visual feedback.

How to run it and its limits

Developed by the Show Lab team at the National University of Singapore, the research paper is available on arXiv, and code has been pushed to GitHub.

However, documentation is currently incomplete, so getting it running in your local environment will require digging through the repository setup manually.

There is also a clear capability trade-off: on basic operating system file management tasks, ShowUI-π only achieved a 13.11% success rate, well below OpenCUA-7B at 99.01% and Gemini-2.5-CUA at 86.14%.

In other words, it is not an all-around OS controller yet, but rather a specialized motor controller built for complex continuous physical tasks.

Why it matters

We think this work highlights an overlooked bottleneck in computer-use agents: semantic understanding is not the same as physical control. Scaling multi-billion parameter foundation models has not solved fine-grained mouse control, because coordinate tokenization simply cannot capture smooth dynamics.

By showing that a 450M model can outperform giant models on video editing timelines and verification puzzles, ShowUI-π makes a strong case for modular agent design. If you are building desktop automation agents, pairing a large planner model with a specialized trajectory expert like this looks like the right blueprint. Just hold off on full deployment until the documentation and tooling mature.