Google TurboQuant: 6x KV Cache Compression Changes AI Inference Economics

The key-value cache is the most expensive part of running a large language model — and until now, nobody had solved it without sacrificing accuracy. At ICLR 2026, Google Research published TurboQuant: a two-stage compression algorithm that reduces KV cache memory by at least 6x, compresses keys to just 3 bits, and achieves an 8x attention compute speedup on NVIDIA H100 GPUs — all with zero accuracy loss and no model retraining required. If the LLM inference cost curve had a breaking point, TurboQuant is it. TechCrunch called it the “Pied Piper moment” for AI compression, and the open-source community agreed: vLLM and llama.cpp integrations landed within days of the paper’s release.

The KV Cache Problem, Explained

Every transformer-based language model — GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro — uses attention mechanisms that require storing key and value tensors for every token in the context window. As the context grows, this key-value cache grows proportionally. Running a model with a 1-million-token context window doesn’t just require a powerful GPU — it requires a GPU with enough RAM to hold the full KV cache for every active request simultaneously.

For a typical 70B parameter model serving a 128K token context, the KV cache alone consumes 20–40 GB of VRAM per concurrent request. Scale that to the million-token context windows that GPT-5.4 and Gemini 3.1 Pro now offer, and the memory requirements become prohibitive for all but the most capital-intensive inference operations. This is why long-context inference is expensive: it’s not the model weights that grow — it’s the cache that balloons with every token.

The industry has tried to solve this in several ways: sliding window attention (limits usable context), KV cache eviction (discards early context tokens), and traditional quantization (compresses cache to 4–8 bits). None of these solved the problem cleanly. Sliding windows and eviction lose information by design. Traditional quantization to 4 bits or below degrades model quality on long-context tasks — exactly the tasks where the cache is largest and most critical. TurboQuant is the first approach to achieve extreme compression with mathematically provable quality guarantees.

How TurboQuant Works: PolarQuant + QJL

TurboQuant uses a two-stage pipeline that separates the compression problem into two orthogonal subproblems, solving each with a mathematically optimal method.

Stage 1 — PolarQuant: The first stage converts each key vector from standard Cartesian coordinates into polar coordinates, separating magnitude (how large the vector is) from direction (where it points in embedding space). This separation is the key insight. In transformer attention, the directional component dominates the similarity computation — the magnitude matters far less. PolarQuant skips the expensive per-block normalization step that conventional quantizers require, because the polar representation makes the angular distribution of key vectors predictable and concentrated. The result is 3-bit quantization of the directional component with minimal information loss. PolarQuant was presented separately at AISTATS 2026 before being incorporated into TurboQuant as its first stage.

Stage 2 — QJL (Quantized Johnson-Lindenstrauss): PolarQuant alone still leaves small systematic errors. The second stage applies a 1-bit error correction layer using the Johnson-Lindenstrauss Transform — a classical result from theoretical computer science that projects high-dimensional data into lower dimensions while preserving pairwise distances. Google’s contribution is the “Quantized” extension: the residual quantization error is projected into a lower-dimensional space and each value is reduced to a single sign bit (+1 or -1), eliminating systematic bias in attention score calculations at essentially zero additional memory overhead.

Together, PolarQuant handles the bulk compression and QJL corrects the residual error. The combination achieves something that neither technique could accomplish alone: near-optimal compression with provable quality guarantees that are distribution-free — meaning they hold regardless of the input data.