Search papers, labs, and topics across Lattice.
This paper introduces EpiKV, a novel cache eviction method that utilizes an "epiphany score" to evaluate token importance without relying on the attention matrix, thus overcoming the limitations of existing methods that hinder efficient deployment in long reasoning tasks. By directly measuring changes in the model's internal representation during the forward pass, EpiKV achieves significant scalability, allowing for a 16x increase in feasible context length compared to traditional attention-based scoring methods. The results demonstrate that EpiKV matches or exceeds the performance of leading attention-based methods while operating at up to 2.8 times the speed, making it a compelling solution for real-world applications.
EpiKV achieves 72% accuracy on MATH-500 with a 4096-token cache, rivaling the best attention-based methods while dramatically improving inference speed.
As reasoning models emit chains of thought tens of thousands of tokens long, KV cache increasingly becomes a deployment bottleneck. Existing cache eviction methods rank tokens by attention weight, which is a noisy importance proxy in long reasoning traces, and prohibits the use of fused kernels in production inference by forcing the model to materialize the attention matrix. In this work, we instead score tokens with a metric we term the epiphany score: the change in the model's internal representation, read directly from the forward pass with no attention matrix and negligible extra state. Our resulting cache eviction method, EpiKV, requires no training, classifier, or custom kernel, and can be used directly in FlashAttention inference stacks unchanged -- scaling to a 16x longer feasible context than attention-based scoring. upper-mid layers negatively) and remove a positional trend with a causal rolling z-score. At a 4096-token cache EpiKV reaches 72% on MATH-500, matching the strongest attention-based baseline (ThinKV 71%, H2O 67%); a lag-normalized KV variant reaches 37% on AIME-2024 at 8192 tokens against the best of them (33%), at up to 2.8x the speed.
CMU ML