Runway turns video generation into a playable game

For years, AI video has been a one-way mirror: you type a prompt, wait a while, and get back an unchangeable five-second MP4. You could watch, but you couldn't touch.

Runway wants to change that. On September 3, the company unveiled GWM Worlds 2, a real-time world model that turns continuous video and audio generation into a playable, interactive simulation. It continuously streams 720p video at 24 fps alongside synchronized 48,000 Hz audio.

Even better, sessions have no preset length. Unlike rigid clips, this world keeps rendering as long as you keep playing.

WASD your way through AI video

Building on December's GWM Worlds, this second iteration relies on an input framework Runway calls WorldPrompt. The idea splits reality into two buckets: what stays persistent and what changes over time.

A "genesis prompt" and a starting image lay down the persistent foundation—materials, lighting, soundscapes, and physical rules like gravity and collision. On top of that sits an event stream of free-form text actions and continuous per-frame camera movement.

In practice, it handles like a video game. You can navigate environments in first- or third-person, driving, walking, or riding through the scene. Camera and character motions are decoupled, meaning you can look around freely while running forward.

Because typing out full sentences mid-game is impossible, Runway's real-time demo binds text actions to keyboard and mouse controls. Pressing W commands the character forward; a left click throws an object. Characters can carry on live conversations with synced lip movements, and players can manipulate objects, like flicking on a lamp or torching a stage. Multiplayer is supported too: via LiveKit streaming, different players can control different avatars, or one can act as the "director" shaping the weather.

How it stays real-time

Under the hood, GWM Worlds 2 is an autoregressive diffusion model handling both video and audio. While standard bidirectional video models produce rich detail, they are far too slow for live play. Runway post-trained its base model into an autoregressive architecture that predicts the world frame-by-frame.

Each generation step conditions on global anchors (the genesis prompt and first frame), current camera and text inputs, and a sliding KV-cache window of recently generated frames. Older frames get evicted as you move along, keeping compute manageable.

You don't even have to start from scratch: the system can prefill from an existing eight-second video clip and pick up the simulation from where it left off, keeping aesthetics and audio consistent.

The catch? It's strictly a research preview. Speed still comes at the expense of fidelity. If you whip the camera around too quickly, geometric structures and fine textures degrade noticeably. Long-term memory also remains imperfect, and you can't feed in image references mid-session.

Why it matters

Shifting AI video from passive playback to continuous interactive simulation changes the entire playing field.

For game creators and virtual world builders, this represents a glimpse into a post-engine future. GWM Worlds 2 isn't ready to replace Unreal Engine—the artifacts during rapid camera pans make that clear—but it offers a tantalizing preview of dynamic virtual worlds generated without manual 3D assets. For robotics and embodied AI researchers, it provides a flexible simulator where agents can test navigation and object manipulation directly from text goals.

We think Runway made a clever trade-off here. Instead of chasing Hollywood-grade 4K offline rendering, they prioritized latency and infinite session length. Binding pre-authored prompts to hotkeys neatly sidesteps the friction of human typing lag.

Real-time neural generation is still in its infancy, but GWM Worlds 2 proves that interactive AI video isn't just a research concept anymore—it is playable today.