Search papers, labs, and topics across Lattice.
This paper addresses the issue of structural-role bias in long-context models by analyzing the performance of attention-based KV cache eviction methods, particularly in schema-dense input streams like nested JSON. The authors find that the current method disproportionately retains non-content tokens, leading to a significant drop in exact-match accuracy due to a degraded signal-to-noise ratio. By implementing a role-conditional allocation strategy that suppresses KEY tokens, they achieve a notable improvement in accuracy while maintaining low computational overhead, closing 63-98% of the performance gap at sub-20% budgets.
Suppressing structural KEY tokens in KV cache eviction can dramatically enhance LLM accuracy, recovering up to 98% of lost performance with minimal computational cost.
Attention-based KV cache eviction (H2O and its descendants) compresses the memory-constrained state of a long-context model by ranking tokens on accumulated attention mass, treated here as signal energy, and keeping the heaviest. On schema-dense input streams such as nested JSON, this score acts as a non-stationary filter that disproportionately retains noise: a non-content sink role (delimiters or whitespace) carries an order of magnitude more energy than any content role, and structural KEY tokens are over-retained at roughly 1.8x the rate of the answer-carrying VALUE tokens, collapsing exact-match accuracy from 88% to 0% at a 5% budget as the signal-to-noise ratio of the retained state degrades. A counterfactual experiment establishes that suppressing KEY tokens is the best deployable filter. Our retraining-free, role-conditional allocation over SnapKV's windowed score, governed by a single tuned hyperparameter, closes 63-98% of the H2O gap at sub-20% budgets and, at higher budgets, modestly matches or exceeds full-cache accuracy -- a small, seed-sensitive denoising effect (borderline significant at B=0.50; not distinguishable from zero at B=0.30 over four seeds). A 15 MB linear role probe supplies these labels at negligible inference cost, though matching parser-level downstream accuracy remains open.
