Titans introduces long-term neural memory beyond 2M tokens
Researchers just introduced Titans, a new model architecture that pairs attention-based short-term memory with a neural long-term memory module that learns at test time. It scales past a 2 million token context window while keeping training and inference fast, outperforming standard Transformers on needle-in-haystack tasks.