Google squeezes LLM memory with TurboQuant
Google Research just unveiled TurboQuant, a new quantization algorithm that shrinks key-value caches down to 3 bits with zero accuracy loss. By ditching the traditional memory overhead of quantization constants, it cuts KV memory by at least 6x and runs up to 8x faster on H100 GPUs. Why it matters: running massive long-context models just got substantially cheaper and faster.