Search papers, labs, and topics across Lattice.
This paper benchmarks quantization against rank reduction for compressing the KV cache in transformer inference across a range of model sizes and architectures. The key finding is that quantization consistently and significantly outperforms rank reduction at matched storage budgets, even when combined in hybrid approaches. The authors attribute this to the discrete failure mode of rank reduction under softmax attention, where removing dimensions can drastically alter attention patterns, while quantization introduces bounded noise that preserves score ordering.
Forget throwing away dimensions – quantizing your KV cache gives you way better compression for the same memory footprint, and the math proves why.
We compare two strategies for compressing the KV cache in transformer inference: rank reduction (discard dimensions) and quantization (keep all dimensions, reduce precision). At matched storage budgets across five models (124M-14B, MHA and GQA), we find that quantization consistently outperforms rank reduction by 4-364 PPL depending on model and compression level. The gap persists even when rank reduction is combined with quantization in hybrid baselines, and it grows with GQA aggressiveness. On LAMBADA, INT4 matches FP16 accuracy (+0.23 PPL on Mistral 7B, +0.58 on GPT-2) while rank-32 at identical storage collapses to 0.4%. We trace this gap to a structural asymmetry: under softmax attention routing, removing a dimension can flip which token is attended (a discrete failure), while quantization noise is bounded and typically preserves score ordering. We formalize this via a perturbation result showing projection damage exceeds quantization damage by 3 x 2^(2b) per direction under the softmax Fisher metric. A basis ablation confirms the finding is basis-independent (spread <0.4 PPL), establishing that the advantage comes from preserving dimensions, not from a better coordinate system. Joint K+V INT4 quantization achieves 75% total KV reduction at only +0.18 PPL on Mistral 7B.
