Search papers, labs, and topics across Lattice.
This paper introduces SeKV, a resolution-adaptive semantic KV cache designed to optimize memory usage for long-context large language model (LLM) inference. By organizing context into entropy-guided semantic spans and leveraging a GPU-CPU memory hierarchy, SeKV allows for token-level reconstruction without the need to retain a full KV cache in GPU memory. The approach demonstrates a 5.9% improvement over existing semantic compression methods while significantly reducing GPU memory requirements by 53.3% at a 128K context length.
SeKV achieves a remarkable 5.9% performance boost in long-context LLM inference while slashing GPU memory usage by over half.
Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.
Microsoft Research