Open-source RotorQuant beats Google at KV cache compression
A new open-source algorithm called RotorQuant just outperformed Google's TurboQuant across every metric. By swapping heavy global transforms for compact Clifford rotators, it hits 10x KV cache compression while boosting prefill speed by 5.3x and decoding by 28%. Why it matters: massive 128K context windows that previously required multiple enterprise GPUs can now fit onto a single 24GB card.
RotorQuant is an open-source KV cache compression tool that slashes LLM context VRAM usage by 10x while speeding up prompt prefill by 5.3x.
Anyone running local LLMs knows the pain of long contexts. At FP16, an 8B model running a 128K context burns roughly 4.6 GB of VRAM just on the KV cache. Scale that up to a 32B model, and you suddenly need multiple GPUs.
How it beats Google's TurboQuant
Google set the standard at ICLR 2026 with TurboQuant, which rotated KV vectors before quantizing them to 3-bit. But TurboQuant relied on a global Walsh-Hadamard Transform that required over 16,000 operations per 128-dimensional vector, creating a massive compute bottleneck.
RotorQuant realized attention vectors live on low-rank manifolds and do not need a full-rank transform. Instead, it breaks vectors into tiny blocks of three numbers and rotates them independently using Clifford rotators from 3D geometric algebra.
That drops the math from 16,000 operations down to around 200, while slashing the transform's parameter count by 44x. Surprisingly, rotating these small blocks also preserves directional features better than global scrambling.
On Llama 3.1 8B at 3-bit quantization, RotorQuant beats TurboQuant across the board. Perplexity improves from 7.07 to 6.91, decode speed jumps 28% from 93 to 119 tokens per second, and prefill speed leaps 5.3x from 722 to 3,822 tokens per second.
In practice, loading a 10,000-token prompt drops from 14 seconds down to under 3 seconds. It also offers an asymmetric mode that compresses only the K cache by 5x with almost zero quality loss.
How to run it
The project is available as a drop-in fork of llama.cpp, meaning no model retraining and no format conversions.
It works out of the box on both CUDA and Apple Silicon. You simply clone the repository, check out the planarquant branch, and build with standard cmake flags for CUDA or Metal.
Why it matters
If you run local document Q&A, codebase search, or agent workflows, prefill latency is usually your biggest bottleneck, and RotorQuant eliminates it.
It proves that clever geometric math can beat brute-force matrix transforms. We think this approach will quickly make its way into upstream llama.cpp and vLLM, making massive 128K contexts on 32B models practical on a single 24GB consumer GPU.
Sources (1)
- themenonlab.blog(opens in a new tab)Originalthemenonlab.blog