Google squeezes LLM memory with TurboQuant

Fans of HBO’s Silicon Valley remember Pied Piper, the scrappy startup that upended the tech world with an impossibly good, lossless compression algorithm.

Now, Google Research seems to have built the real thing for artificial intelligence.

Google researchers Amir Zandieh and Vahab Mirrokni just unveiled TurboQuant, an ultra-efficient quantization algorithm designed to dramatically compress large language models and vector search engines. The research will be presented at ICLR 2026 and AISTATS 2026.

Ditching coordinates to kill memory overhead

When AI models process massive documents, their working memory quickly gets clogged by the key-value (KV) cache. Think of this as the model's digital cheat sheet, storing recent context so it does not have to recalculate everything from scratch.

Traditional vector quantization compresses this data, but it carries a hidden tax. Most methods require calculating and storing full-precision constants for every small data block. This overhead can add 1 or 2 extra bits per number, eating away at the compression gains.

TurboQuant fixes this by combining two mathematical techniques: Quantized Johnson-Lindenstrauss (QJL) and PolarQuant.

QJL compresses complex vectors down to a single sign bit (+1 or -1) while preserving core geometric distances, effectively requiring zero memory overhead. It pairs high-precision queries with low-precision stored data to calculate attention scores accurately.

PolarQuant takes an even wilder approach: instead of standard Cartesian coordinates (X, Y, Z), it converts vectors into polar coordinates (a radius and an angle).

Rather than saying "go 3 blocks east and 4 blocks north," it says "head 5 blocks at a 37-degree angle." Because the distribution of angles follows a known, predictable pattern, data can be mapped directly onto a fixed circular grid. That eliminates the need for expensive normalization and wipes out the memory overhead traditional quantization methods struggle with.

6x memory drop and an 8x speedup

Google tested TurboQuant across standard long-context benchmarks like LongBench, Needle In A Haystack, and RULER using Gemma and Mistral.

The results were striking. TurboQuant compressed the KV cache down to just 3 bits without any training or fine-tuning, maintaining zero accuracy loss. In needle-in-a-haystack retrieval tests, it delivered perfect downstream performance while shrinking KV cache memory by at least 6x.

On Nvidia H100 GPUs, 4-bit TurboQuant calculated attention logits up to 8x faster than unquantized 32-bit keys.

It also shone in high-dimensional vector search. Benchmarked against industry standards like PQ and RabbiQ, TurboQuant posted superior recall rates without needing massive codebooks or dataset-specific tuning.

The internet immediately dubbed it the real-world Pied Piper. Cloudflare CEO Matthew Prince went further, calling it Google's "DeepSeek moment" in a nod to the recent wave of efficiency-driven AI breakthroughs.

Why it matters

We think TurboQuant tackles one of the most painful choke points in AI today: the cost of long-context inference.

For developers serving millions of tokens, memory footprint is the primary bottleneck capping concurrency. Slashing KV cache requirements by more than 80% means teams could run significantly larger context windows on far fewer GPUs, making long-context applications viable on smaller hardware setups.

For the broader industry, it highlights a welcome shift away from brute-force compute toward elegant mathematical foundations. Using polar transformations and JL projections to dodge normalization overhead is a clever engineering choice that rivals will likely rush to replicate.

That said, let's keep expectations grounded. TurboQuant remains a lab breakthrough that hasn't yet been deployed in production at scale. It also targets inference cache and vector retrieval; it does not alleviate the massive VRAM appetites required during model training.

Still, achieving 3-bit compression with zero fine-tuning is an undeniable technical feat. If real-world deployments match these lab numbers, it could quickly become table stakes for enterprise inference architectures.