Google’s new Gemini model clicks and scrolls for you

AI models are great at generating code, but the moment you ask them to log into a clunky web portal, pick from a dropdown, and hit submit, they hit a wall.

Google is tackling that interface barrier with the release of its new Gemini 2.5 Computer Use model. Built on the visual reasoning chops of Gemini 2.5 Pro, the specialized model powers agents that can see, click, scroll, and type across software interfaces just like a human.

The model is now available in public preview via the Gemini API in Google AI Studio and Vertex AI.

Closing the browser loop

The engine behind this is a new API tool called computer_use, designed to run in a continuous feedback loop.

The setup is simple: the client feeds the model a task prompt, an environment screenshot, and a log of recent actions. The model analyzes the screen and returns a specific function call, like clicking a button or typing into a form field.

Once the client executes that action, it takes a fresh screenshot alongside the current URL and sends them right back into the model. The loop continues until the job is done, an error occurs, or the session gets halted.

Developers can also set guardrails to exclude certain UI actions or require manual user confirmation before high-stakes steps, like hitting a checkout button.

Google noted that the model is heavily optimized for web browsers and shows strong promise on mobile interfaces, though it is not yet tuned for desktop OS control.

Low latency and built-in guardrails

Latency has always been the Achilles' heel for computer-use agents. Google claims Gemini 2.5 Computer Use beats competing options across multiple web and mobile benchmarks, citing evaluations by Browserbase on the Online-Mind2Web harness that show lower latency alongside high accuracy.

Inside Google, the tech is already running in production. Teams use it for automated UI testing to speed up software development, and variants power tools like Project Mariner, Firebase Testing Agent, and agentic workflows in Google Search AI Mode.

Letting AI loose on live websites brings serious risks, from prompt injections to scam pages. Google claims it baked defenses directly into model training and added an out-of-model, per-step safety service that checks each proposed action before execution.

Developers can also enforce hard rules preventing the agent from tackling sensitive tasks, such as bypassing CAPTCHAs, tampering with system integrity, or touching medical systems.

Why it matters

AI agents are finally moving past conversational chat boxes and getting their hands on digital tools.

For developers and automation engineers, this is a massive shortcut. Endless enterprise workflows remain trapped behind outdated web interfaces without APIs. An agent that can visually navigate forms and messy dashboards means complex multi-step automations can finally run without brittle custom scraping scripts.

Focusing squarely on the web rather than full desktop control feels like a smart tactical move from Google. Most business workflows live inside a browser, where security sandboxing is much easier to manage than granting an AI full rein over an operating system.

That said, real-world websites are messy. Aggressive bot blockers, sudden popups, and dynamic rendering often break autonomous loops. Whether Gemini can maintain high reliability across messy real-world web sessions without constant human babysitting remains the ultimate test.