Search papers, labs, and topics across Lattice.
This paper introduces QEvict, a novel three-tier KV-cache management scheme that addresses the limitations of traditional token eviction policies in autoregressive large language models. By allowing for recoverable eviction, QEvict dynamically updates window importance based on cumulative attention scores, enabling previously discarded tokens to be reactivated when they become relevant again. The method significantly enhances long-context decoding performance, demonstrating improved information retention and reduced missed attention across various benchmarks compared to existing approaches.
Recoverable eviction can drastically reduce missed attention and improve information retention in long-context decoding, outperforming traditional methods.
Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache. A dominant line of work reduces this footprint by evicting tokens that appear unimportant under attention-derived scores. However, such policies make an implicit irreversible decision: once a token is evicted, it cannot become useful again. We show that this assumption is brittle during decoding. Token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states that later receive substantial attention under the full-cache model. To characterize this behaviour, we introduce Future Missed Mass and Global LIR, two diagnostics that measure future attention assigned to discarded states and the reactivation of historically inactive regions. We propose QEvict, a three-tier KV-cache management scheme that replaces binary retain-or-delete eviction with recoverable eviction. QEvict maintains high-confidence windows in full precision, stores intermediate windows in a quantized recoverable tier, and deletes only the lowest-confidence windows. During decoding, cumulative attention scores update window importance and when a quantized window becomes important again, it is dequantized and promoted to the full-precision. Under a fixed memory budget, this design preserves broader historical context while retaining exact full precision for the most important regions. Across long-context understanding, retrieval, and reasoning benchmarks, QEvict consistently improves over representative eviction and quantization baselines, reducing missed attention and improving information retention
