Nvidia's DMS shrinks LLM memory up to 8x without hurting reasoning

The smarter modern reasoning models get, the more GPU memory they eat alive.

Every time a model generates a long chain of thought, its temporary memory cache balloons. Nvidia researchers just unveiled a fix called Dynamic Memory Sparsification (DMS), a technique that cuts KV cache memory footprints by up to 8x without hurting reasoning accuracy.

Teaching models to forget

Until now, squeezing memory out of large language models relied on clumsy tricks. Engineers used sliding windows to blindly delete older words or paged data back and forth to slower memory. The result was almost always the same: either the model forgot critical facts and got dumber, or response latency tanked.

Nvidia took a different path by retrofitting existing models like Llama 3 and Qwen 3 to manage their own memory. Instead of training from scratch, DMS repurposes existing attention neurons to output keep or evict signals for each token. Piotr Nawrot, a senior deep learning engineer at Nvidia, noted that teams can freeze the base weights and retrofit a Qwen3-8B model within hours on a single DGX H100 system, similar to running a LoRA.

The real secret sauce is delayed eviction. When a token is flagged for deletion, the system keeps it in a temporary window for a few hundred steps. This gives the model time to absorb context before the data is wiped from VRAM.

Under identical memory limits on the AIME 24 math benchmark, a Qwen-R1 32B model equipped with DMS beat standard models by 12 points. Nvidia has already made the technique available inside its Model Optimizer framework.

Why it matters

Inference cost is quickly replacing pre-training cost as the enterprise AI bottleneck. When models generate thousands of hidden thinking tokens, serving real users becomes unsustainably expensive.

Being able to handle 800 reasoning threads on infrastructure that previously supported 100 completely changes the unit economics of agentic AI. It lets developers unlock deeper reasoning loops and multi-path exploration on hardware they already own, rather than constantly buying more VRAM.

What makes this approach clever is that it is non-destructive and practical. You do not need massive clusters or custom architectures; an afternoon of lightweight fine-tuning turns an off-the-shelf open-source model into a memory-efficient engine. For any team running heavy chain-of-thought workloads, this looks like one of the most practical efficiency upgrades to land this year.