Why massive context windows could unlock true AI continual learning

When people hear about longer context windows, the default reaction is usually something along the lines of "cool, I can drop ten Harry Potter books into Gemini 2.5 Pro." But according to a new analysis from Epoch AI, treating context purely as a massive digital filing cabinet misses the entire point.

Context windows have been expanding at a staggering 30x per year. If that trajectory holds even partially, moving toward tens of millions or billions of tokens will fundamentally rewire how models learn and reason.

Solving the continual learning problem

One of the biggest roadblocks in modern AI is that models basically have amnesia. Their weights are frozen after training, meaning they cannot organically acquire new tacit knowledge from past mistakes over months of real-world use.

Epoch AI points out that this is largely a context bandwidth bottleneck. If an AI agent operates a computer using screen captures, a 1-million-token window only covers about 30 minutes of activity. Scale that to 10 million tokens, and you get six hours. Jump to 10 billion tokens, and you suddenly have eight months of uninterrupted work history.

With that kind of headroom, in-context learning becomes a viable substitute for fine-tuning. Researchers like Andrej Karpathy and Cursor's Aman Sanger have already sketched out similar ideas. A model could even leverage scheduled "sleep-time compute" during off-hours, running RL across its own interaction history and compressing tacit habits into an optimized KV cache.

Fuelling the next phase of RL scaling

Massive context windows are also becoming indispensable for scaling reinforcement learning at test time. Modern reasoning models are generating longer and longer chains of thought to crack complex challenges.

In frontier scientific research, feedback signals are notoriously sparse. An agent might have to spend weeks proposing hypotheses and running code before knowing whether an experiment worked. Evaluating those sprawling rollouts requires verifiers that can ingest enormous reasoning traces, much like the Universal Verifier infrastructure OpenAI built to train its frontier models.

Long contexts also enable the synthetic generation of rich simulation environments, an approach already seen in models like Kimi K2 using MCP servers. However, wall-clock time is emerging as a hard physical wall. As OpenAI researcher Noam Brown noted on the Latent Space podcast, when an experiment takes hours or weeks of model thinking to complete, research iteration slows down to a crawl. There is also the reality of inference costs, which led Google DeepMind to hold back the public release of its 10-million-token window.

Why it matters

For builders and enterprise teams, this shifts the perspective on what agents need to succeed. Current agent frameworks often derail over extended tasks—as seen in benchmarks like Vending Bench where models quickly run vending machines into financial ruin—not because the models are fundamentally incapable, but because their operational scratchpad is too cramped. Waiting for cheaper long-context inference will likely be far more effective than building overly complex prompt-chaining hacks today.

Across the broader landscape, this dynamic moves the competitive front line from massive pre-training runs to ultra-long inference efficiency. The frontier will belong to whoever can drive down the cost per token of reasoning across hours-long Horizons.

We think framing long context as an engine for continuous learning and RL verification is spot-on. But there is a glaring catch: raw token capacity does not guarantee coherence. If models still over-index on early mistakes or wander off track after a few thousand tokens, stuffing billions of tokens into memory might just give AI agents more room to confidently fail.