Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of key-value (KV) cache eviction strategies in sparse-attention models by analyzing the optimal subset of keys to retain and their associated attention weights. The findings reveal that retaining the largest weights is nearly optimal, closing only a median gap of 2-5% to full attention, suggesting that the differences in performance among eviction methods stem from other factors. The proposed ContourKV allocator, which utilizes a dropped-mass statistic, outperforms existing methods in 93 out of 160 comparisons while maintaining competitive memory efficiency.
Retaining the largest attention weights in KV-cache eviction is nearly optimal, challenging the perceived complexity of selection strategies.
Every deployed sparse-attention or KV-cache-eviction rule keeps a subset of the keys, discards the rest, and renormalizes the attention weights over the kept set. Enumerating the exact best subset under that constraint on $168{,}192$ attention rows from five models shows that keeping the largest weights is already near-optimal, since the best subset closes only a median $2$ to $5\%$ of the remaining gap to full attention. If selection closes this little, published margins between eviction methods must come from elsewhere, so we measure the bytes each method holds. In the shared evaluation pipeline, the strongest query-agnostic methods hold the full cache because their per-head selections are stored as masks, and only ragged per-head storage frees that memory. Enforcing a nominal budget on one fixed selection costs $14$ to $62$ benchmark points. We trace an $87.6$-point retrieval margin to rankings computed while the question is visible. ContourKV, a training-free allocator built from the dropped-mass statistic, wins $93$ of $160$ paired comparisons against that state of the art and loses $22$ at the byte count of the budget-enforcing baselines, and it ties the strongest of them.
