Search papers, labs, and topics across Lattice.
This paper introduces Value-aware Stochastic KV Cache Eviction (VaSE), a method designed to enhance the accuracy of reasoning models while addressing the memory and compute bottlenecks associated with long output chains. By identifying that certain high-magnitude value states can lead to catastrophic failures when evicted, the authors implement a stochastic approach to eviction that increases cache diversity and protects these critical states. The results demonstrate that Qwen3 models utilizing VaSE achieve higher average accuracies across six reasoning tasks compared to state-of-the-art selection methods, even with 4x KV cache compression, effectively balancing efficiency and accuracy.
Stochastic eviction can boost reasoning model accuracy by protecting critical value states while achieving significant memory savings.
Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.
