Search papers, labs, and topics across Lattice.
The paper introduces InnerQ, a hardware-aware quantization scheme for compressing the KV cache in large language models to reduce memory footprint and improve decoding latency. InnerQ employs group-wise quantization along the inner dimension of the KV cache matrices, enabling efficient dequantization and scale factor reuse on GPUs. The method incorporates hybrid quantization, high-precision windows, and per-channel key cache normalization to preserve accuracy under aggressive compression, achieving up to 22% speedup over prior KV cache quantization methods on Llama models.
By quantizing KV caches along their inner dimension, InnerQ achieves up to 22% speedup in LLM decoding compared to prior art, without sacrificing accuracy.
Reducing the hardware footprint of large language models (LLMs) during decoding is critical for efficient long-sequence generation. A key bottleneck is the key-value (KV) cache, whose size scales with sequence length and easily dominates the memory footprint of the model. Previous work proposed quantization methods that are focused on compressing the KV cache while maintaining its information. We introduce InnerQ, a hardware-aware KV-cache quantization scheme that lowers decode latency without sacrificing accuracy. InnerQ applies group-wise quantization while grouping the cache matrices over their inner dimension. Unlike previous work that group over the outer dimension, InnerQ aligns dequantization with the vector-matrix multiplication and enables scale factor reuse across GPU compute units. This reduces memory accesses and accelerates dequantization, yielding up to $22\%$ speedup over previous work and up to $88\%$ over half-precision vector-matrix multiplication. To preserve fidelity under aggressive compression, InnerQ incorporates (i) hybrid quantization, selecting symmetric or asymmetric quantization per group based on local statistics; (ii) high-precision windows for both the most recent tokens and the attention sink tokens to mitigate outlier leakage; and (iii) per-channel normalization of the key cache, computed once during prefill and folded into the query to avoid runtime overhead. Our evaluation experiments on Llama models shows that InnerQ maintains a few-shot GSM8K performance comparable to non-quantized KV caches and surpasses prior KV cache quantization methods.
