Search papers, labs, and topics across Lattice.
This paper introduces Gain-Shape Residual Quantization (GSRQ), a novel approach that enhances Key-Value (KV) cache memory efficiency for Large Language Models (LLMs) by addressing the limitations of traditional $\ell_2$ $K$-means codebook learning. By replacing standard $K$-means with Gain-Shape $K$-means (GSKM), the method improves directional fidelity and reduces centroid shrinkage, leading to better angular alignment and reduced distortion. Experimental results on LLaMA-3-8B demonstrate that GSRQ significantly boosts accuracy on LongBench tasks, achieving an average accuracy increase from 11.34 to 33.54 at 1-bit quantization, outperforming existing baselines.
GSRQ achieves a remarkable 22.20 percentage point accuracy gain on LongBench tasks, revolutionizing KV cache efficiency for LLMs at sub-1-bit quantization.
The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory. Vector Quantization (VQ), particularly Residual Quantization (RQ), is a promising approach for pushing KV cache storage toward the sub-1-bit regime by progressively encoding residuals with small codebooks. However, most VQ methods still rely on standard $\ell_2$ $K$-means as the core codebook-learning primitive. We identify a subtle high-dimensional issue of this primitive: Euclidean centroid averaging can induce centroid shrinkage, which weakens the angular alignment term in the $\ell_2$ distortion and makes directional preservation harder. To address this issue, we propose Gain-Shape $K$-means (GSKM), a drop-in replacement for $K$-means that improves directional fidelity while matching, and in some regimes improving, $\ell_2$ distortion. We then build Gain-Shape Residual Quantization (GSRQ) by incorporating a weighted extension of GSKM into an RQ pipeline. On LLaMA-3-8B, GSRQ substantially improves over strong KV cache quantization baselines across bit rates. At 1-bit, it improves the average accuracy across LongBench tasks from 11.34 to 33.54, a gain of 22.20 percentage points over VQLLM.
