Code World Model uses coding agents to give AI video persistent memory

If you have spent any time playing with AI video models, you know their biggest flaw: they have goldfish memory. A character walks through a wooden door, turns around, and the door is suddenly a brick wall. Pick up an apple, and two seconds later the model hallucinates it into thin air. Traditional video models are simply guessing the next cluster of pixels based on visual patterns. They have no concept of permanence, physical rules, or consequence.

Researchers published and open-sourced Code World Model, or CWM. Instead of forcing a video generator to dream up both reality and its underlying physics, the project takes a refreshingly pragmatic approach: let an AI coding agent run the logic, and let a video model handle the paint.

Decoupling the game engine from the graphics card

The fundamental premise of CWM is separating world evolution from visual rendering.

At the center of the system sits a coding agent acting as the world brain. When an event happens—say, an object moves or an action is taken—the agent reasons about the consequences, writes executable code, and updates a persistent world state. It behaves exactly like an invisible game engine running in the background.

To bridge this code state with visual generation, the team introduced a proxy representation. This proxy compiles frame-by-frame spatiotemporal constraints—like where objects sit and how the scene is laid out—into a rough proxy video.

Finally, the researchers fine-tuned MiniMax-H3 on paired gameplay and real-world video data. The video model takes the proxy video as a conditioning prompt and renders high-fidelity visual observations. The coding agent ensures the rules never break, while MiniMax-H3 makes the scene look visually rich.

Long horizons and style changes that keep state

The project demos highlight just how much coherence improves once code takes the wheel.

In long-horizon generations, the virtual environment holds together over extended runs. One demo even features a real-time minimap in the corner tracking every object in the scene, proving that the world state remains rock-solid regardless of camera angles.

Because the underlying logic is decoupled from visuals, the system can even execute periodic style changes. You can switch the visual rendering from realistic to stylized while preserving the exact layout, objects, and historical events.

The codebase is already out on GitHub with setup instructions. However, running it is not lightweight. The authors conducted their experiments on an NVIDIA H800 GPU with 80GB of VRAM, though quantized community variants could eventually lower that hardware floor.

Why it matters

We think Code World Model points toward the right architecture for interactive AI worlds. Expecting a pure diffusion model to spontaneously learn spatial math and object permanence across thousands of frames has always been an uphill battle. Language models already excel at deterministic logic and code execution. Combining code-based state tracking with generative rendering plays directly to both models' strengths.

For game developers and simulation researchers, this opens up a viable path toward truly dynamic, agent-driven environments. Instead of hard-coding every interaction tree or accepting hallucinated video mush, you get persistent logic paired with adaptive graphics.

That said, high VRAM requirements and rendering latency mean you will not be playing a real-time AAA title built on this tomorrow. But as a proof of concept, CWM makes a convincing argument: if AI is going to build worlds we can actually explore, it needs an engine written in code, not just a stream of lucky guesses.