Search papers, labs, and topics across Lattice.
This paper introduces KVpop, a novel key-value cache eviction method that learns to optimize memory usage during autoregressive decoding by directly supervising the keep-or-drop decision based on future token utility. By employing a future-attention target and a delayed memory-based scorer, KVpop achieves impressive performance retention, maintaining 98% of full-attention performance with 75% cache compression and 97% at 88% compression on specific reasoning tasks. The results demonstrate that this approach significantly reduces memory costs while preserving the quality of output, outperforming traditional eviction methods.
Learning to prune key-value caches based on future token utility allows KVpop to achieve up to 88% compression while retaining nearly full performance.
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.
