ShowUI-Aloha learns complex desktop tasks from a single recording

The hardest part of getting an AI to control your desktop isn't vision or text comprehension—it's that general-purpose models easily lose their mind the moment they face real-world OS complexity. Ask an agent to edit a project file, and it often gets stuck in an infinite loop searching folders or randomly clicking through nested menus.

ShowLab just took a much more practical route with ShowUI-Aloha: if prompting doesn't work, just show it how the job is done.

Learn the workflow, don't just memorize the pixels

ShowUI-Aloha is an open-source computer-use agent built for real Windows and macOS environments. Instead of relying purely on complex natural language prompts, it learns directly from human demonstrations. You start the recorder, complete a task yourself, and let the system capture your screen frames, mouse movements, and keystrokes.

Crucially, it isn't just replaying recorded clicks. The Learner module distills low-level OS interactions into high-level semantic traces—abstract concepts like "open browser," "fill form," or "save slide." When a new instruction arrives, the Planner adapts that structural workflow to handle variations.

Demonstrate how to book a flight once, and it understands how to book a completely different itinerary. Show it how to transpose a matrix in Excel, and it applies that sequence to fresh spreadsheets. On the grueling OSWorld benchmark of 361 multi-step desktop tasks, Aloha completed 217 end-to-end, scoring a strict 60.1% success rate and comfortably beating baseline models.

Outperforming commercial agents on the dirty work

The team ran head-to-head comparisons against strong commercial agents, and the differences are telling.

In a local GitHub repository update task, commercial agents failed to infer that repos were stored under Documents/GitHub, falling into endless folder search loops. Aloha simply followed the human-taught directory structure and finished the job.

In PowerPoint, commercial models misidentified a tiny ribbon icon and applied an orange background instead of yellow, cascading into unrecoverable UI errors. In Excel, commercial agents repeatedly failed to locate the Transpose option buried inside paste menus. Aloha replicated the demonstrated sequences cleanly in both cases.

The project is fully open source under the MIT License, providing recorder binaries for Windows and macOS. Running the execution pipeline does require bringing your own VLM API key.

Why it matters

We think ShowUI-Aloha makes an exceptionally smart compromise. Instead of demanding that a foundation model solve obscure desktop interfaces from zero context, it uses a single human recording as an anchor to eliminate the cost of trial-and-error exploration.

For power users and independent developers, this opens up practical desktop automation for workflows where writing traditional scripts is simply too painful—like repetitive batch edits in presentation software or moving data across legacy desktop tools. You don't have to code; you just do the work once.

The catch is robustness. Aloha's success hinges heavily on the initial workflow structure. If an application rolls out a major UI redesign or throws unexpected popups that were never demonstrated, recovery could be fragile. The roadmap already lists fine-grained element targeting, drag-based text editing, and Linux support as active priorities.

Still, shifting from brute-force prompt engineering to demonstration-based trace reuse feels like the right highway toward real, usable desktop AI agents.