Search papers, labs, and topics across Lattice.
Jacap establishes an information-theoretic framework for KV cache eviction by modeling attention as a nonlinear Gaussian communication channel rather than relying on empirical heuristics. Using a first-order Taylor expansion of the attention map, the method derives the Jacobian Information Capacity to jointly capture query relevance, softmax sensitivity, and token diversity. Across diverse architectures and long-context benchmarks, Jacap consistently outperforms existing pruning baselines, exhibiting especially high robustness in aggressive compression regimes.
Heuristic KV cache pruning routinely fails under heavy compression because it ignores softmax curvature, but treating attention as a nonlinear Gaussian channel reveals exactly which tokens preserve information capacity.
Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models. However, existing policies predominantly rely on empirical heuristics, lacking a rigorous characterization of token utility under the inherently nonlinear softmax attention mechanism. In this work, we rethink KV cache eviction through the lens of local information geometry, modeling the attention process as a nonlinear Gaussian communication channel. By performing a first-order Taylor expansion of the attention mapping, we derive the Jacobian Information Capacity, a novel objective that explicitly captures query relevance, softmax sensitivity, and structural diversity. Guided by this theory, we introduce Jacap, a capacity-aware eviction method that utilizes softmax-aware importance weighting and statistical leverage scores for subset selection. Extensive experiments across diverse architectures and benchmarks demonstrate that \textsc{Jacap} delivers superior performance in most scenarios, particularly in high-compression regimes.
