Google DeepMind's SIMA plays 3D video games through plain English instructions

Google DeepMind used to build gaming AIs to humiliate human champions. Systems like AlphaStar mastered StarCraft II through inhuman APM and frame-perfect micro. But they were hyper-specialized savants: change the rules or drop them into a different map, and they were completely helpless.

Now, DeepMind is flipping the script. The lab just unveiled SIMA (Scalable Instructable Multiworld Agent), a generalist AI agent built not to chase high scores, but to understand everyday language and follow your lead inside 3D virtual worlds.

Playing games through human eyes

The most impressive part of SIMA is its interface. It does not look at game code, read internal memory, or hook into custom developer APIs.

Instead, it interacts with games exactly like a human player does. It takes in on-screen pixels, listens to natural-language commands, and outputs keyboard and mouse movements. If a human can play it on a monitor, SIMA can theoretically play it too.

To train it, DeepMind partnered with eight game studios across nine commercial titles. The lineup includes Hello Games’ No Man’s Sky, Tuxedo Labs’ Teardown, Coffee Stain’s Valheim, Goat Simulator 3, and Satisfactory.

They also dropped it into four specialized research sandboxes, including a custom Unity environment called Construction Lab, where the agent has to stack blocks to test its intuitive physics.

The training data came directly from people. In one setup, one player gave spoken instructions while another controlled the character. In another, players recorded free gameplay and retroactively labeled the instructions that would prompt each action.

Right now, SIMA can execute roughly 600 basic skills. It handles low-level tasks that take about 10 seconds to finish, like climbing a ladder, turning around, or opening a menu to check a map.

The generalist overtakes the specialist

Training an agent across a wide library of worlds unlocked an unexpected breakthrough: generalists crushed the specialists.

DeepMind found that a SIMA agent trained across all nine 3D games dramatically outperformed agents trained solely on a single game. Even wilder, when dropped into a brand-new game it had never seen before, SIMA performed nearly as well as an agent trained specifically for that environment.

Language is genuinely steering the ship here. In control tests where researchers stripped away text prompts, the agent acted like a wandering bot: it would aimlessly harvest resources or walk around without any coherent goal.

The clear limitation is scope. Ask SIMA to walk over to the spaceship, and it works. Ask it to find copper and build an outpost, and it falls apart. Complex multi-step reasoning is still beyond its reach.

Why it matters

Language models can pass the bar exam, but they cannot click a mouse. They are brains without bodies, stuck behind chat windows.

SIMA matters because video games are the ultimate sandbox for embodied AI. Real-world robotics is slow, dangerous, and expensive. Training an agent inside rich 3D virtual worlds lets AI learn spatial reasoning, physical constraints, and goal execution at zero hardware risk.

DeepMind’s architectural choice here is brilliant. By strictly relying on raw pixels and standard keyboard-mouse inputs, they bypassed the nightmare of proprietary game engines. Any PC game instantly becomes a potential training ground.

For gamers, this is not an esports bot; it is the early prototype of a true NPC companion. Imagine an AI sidekick in No Man’s Sky that you can simply tell to mine some carbon while you survey a planet.

The real test will be horizon length. Ten-second tasks are cute, but real autonomy requires hours-long planning and error correction. If DeepMind can scale SIMA from micro-actions to macro-strategy, the line between video game companion and real-world desktop agent will blur very quickly.