Titans introduces long-term neural memory beyond 2M tokens

Trying to make an AI model read an entire technical manual without hallucinating or slowing to a crawl has always been an expensive engineering nightmare. Standard attention gives you precision but burns compute at quadratic scale, while recurrent networks compress history so hard they forget what they read ten pages ago.

Researchers posted a paper on arXiv introducing Titans, a brand-new neural architecture family. It rethinks model memory from the ground up, pushing context windows past 2 million tokens without blowing up inference latency.

Turning attention into short-term memory

For the past decade, AI architectures have bounced between recurrent hidden states and standard attention. Recurrent networks are fast but lose data; attention catches every dependency but traps models inside fixed-size context windows due to quadratic cost.

Titans tackles this dilemma with a clean division of labor. The researchers treat standard attention strictly as short-term memory, letting it handle precise local dependencies within a limited sliding window.

To handle everything else, they added a dedicated neural long-term memory module. This module literally learns to memorize context at test time, storing historical data and feeding long-past information back into the attention layer whenever it is needed.

Crucially, this long-term memory module still supports fast, parallelizable training while keeping token generation swift. The paper outlines three distinct architectural variants to show different ways these two modules can be wired together.

Scaling past 2 million tokens in the haystack

The dual-memory design paid off across a wide range of benchmarks. In language modeling and common-sense reasoning, Titans consistently outperformed standard Transformers as well as modern linear recurrent models.

Its strengths also carried over into specialized long-sequence domains, showing stronger results on genomics and time series tasks.

Most impressively, Titans scaled past a 2 million token context window. In needle-in-haystack evaluations, where models must retrieve a single hidden fact buried inside massive context, Titans achieved higher retrieval accuracy than the baselines it went up against.

Why it matters

If you build applications that chew through massive codebases, longitudinal health data, or entire legal filings, Titans is a design worth tracking closely. It suggests you might not have to choose between Transformer-level retrieval quality and the speed of linear recurrence.

For the broader AI ecosystem, Titans represents an intriguing philosophical shift. Instead of stretching pure attention until GPU clusters cry, or relying solely on state-space models that struggle with precise recall, it splits cognitive duties into distinct short-term and test-time long-term blocks.

We think this hybrid framing is genuinely smart. The big question moving forward will be real-world production engineering, specifically how dynamic test-time memory handles drift, multi-tenant caching, and state management at scale.