Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To mitigate memory bandwidth bottlenecks in long-context inference, WUSH-KV derives data-adaptive key and value transforms from the second-order statistics of attention matrix operands to minimize quantization error. The method absorbs the value transform directly into model weights, applies the key transform post-RoPE, and guarantees near-optimal reconstruction error under QuEST INT quantization. When integrated into SGLang with percentile-clipped affine quantization, WUSH-KV achieves lower perplexity than existing transform baselines and matches or outperforms state-of-the-art transforms like OSCAR at aggressive 2-bit precision.
Pushing KV cache quantization down to 2 bits without sacrificing downstream accuracy is now viable by rotating key and value matrices according to their joint second-order interaction statistics.
KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck, we introduce WUSH-KV for low-bit KV-cache quantization. It adapts WUSH, which constructs a data-aware transform from the second-order statistics of both factors in a matrix product to reduce quantization error. WUSH-KV uses calibration data to construct separate key and value transforms, with the value transform folded into the model weights and the key transform applied after RoPE. The transforms can be paired with clipped quantizers. For one such quantizer, QuEST INT, we show that, under mild assumptions, the WUSH transform is near-optimal. With this quantizer, WUSH-KV reduces layerwise reconstruction error and achieves the lowest end-to-end perplexity among other tested transforms. For end-to-end evaluation, we integrate WUSH-KV into SGLang using OSCAR-style percentile-clipped affine quantization. At 2-bit, WUSH-KV performs comparably to or outperforms the OSCAR transform across all evaluated models and downstream tasks.