Search papers, labs, and topics across Lattice.
This paper investigates the impact of key-value (KV) cache compression on LLM performance, arguing that current evaluations overlook the role of attention as a routing mechanism. They introduce a physics-inspired framework to analyze KV compression as a perturbation of token-level routing, distinguishing between retention, accessibility, and utilization. Through synthetic tasks, the authors demonstrate a sharp "hallucination safety cliff" near 90% compression, correlated with spikes in Global Eviction Ratio (GER), and identify representational rigidity as a factor limiting compression tolerance.
LLMs can tolerate moderate KV cache compression, but pushing past a 90% threshold triggers a "hallucination safety cliff" linked to a phase transition in semantic reachability.
As context windows in LLMs scale to 100K+ tokens, the key-value (KV) cache becomes the dominant memory bottleneck, with recent methods claiming 80-90% savings and minimal benchmark degradation. We argue these evaluations miss a structural issue: attention is not just storage but routing, and retaining KV pairs does not guarantee semantic accessibility. We propose a physics-inspired view of KV compression as a controlled perturbation of token-level routing, distinguishing retention, accessibility, and utilization. Using synthetic tasks probing multi-entity tracking, disambiguation, coreference, and multi-hop reasoning, we find that moderate compression degrades internal representations with little accuracy loss, revealing redundancy; all models exhibit a sharp hallucination safety cliff near 90% compression, correlated with spikes in Global Eviction Ratio (GER), suggesting a phase transition in semantic reachability; and architectures differ in routing dynamics, with LLaMA showing early consensus and late diversification, and Qwen showing funnel-like late convergence, leading to distinct resilience profiles. Beyond erasure, we identify representational rigidity, where excessive head-level consensus collapses routing flexibility despite token survival. These results suggest sparse token-route structures govern compression tolerance, reframing KV compression as a structural probe of attention geometry and linking long-context scalability to sparsity and the lottery ticket hypothesis in self-attention.
