Search papers, labs, and topics across Lattice.
This paper introduces FreqDepthKV, a novel cache compression method that optimizes KV state storage for long-context LLM inference by factorizing adjacent-layer KV states into shared low-frequency components and sparse high-frequency residuals. The method employs a lightweight online probe to dynamically assign attention heads to different cache modes based on their impact on reconstruction-sensitive attention logits, allowing for adaptive compression without retraining. Experimental results demonstrate that FreqDepthKV maintains high task accuracy across various benchmarks while significantly reducing memory usage and improving decoding throughput, achieving a 3.9x effective compression ratio.
FreqDepthKV achieves a remarkable 3.9x effective compression ratio while preserving high accuracy in long-context LLM tasks, revolutionizing cache management.
Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning. We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits, allowing the compression policy to adapt to prompt structure without retraining. Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets. With a 32k-token prefill window, FreqDepthKV reaches 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods. It also improves decoding throughput to 70.4 tokens/s, reduces TTFT to 2.06 seconds, and lowers peak KV memory to 6.2 GB, achieving a 3.9x effective compression ratio.
