Search papers, labs, and topics across Lattice.
This study investigates the impact of attention substrate on the transition from soft to hard KV-cache eviction in Transformers, specifically comparing sigmoid and softmax attention mechanisms. The authors conduct a controlled experiment with GPT-2-scale models, revealing that while sigmoid attention performs worse as a dense language model, it enables effective KV entry deletion with minimal perplexity change when using learned hard eviction. The findings indicate that the choice of attention normalization significantly influences the efficacy of cache eviction strategies during inference, with sigmoid-gated models outperforming traditional methods in specific scenarios.
Sigmoid attention, while less effective in dense modeling, dramatically improves KV-cache eviction performance, challenging conventional wisdom about attention mechanisms.
Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this soft-to-hard transition. Using GPT-2-scale Transformers trained on OpenWebText, we run a controlled $2\times2\times2$ comparison over attention type, learned gating, and positional encoding. Although sigmoid attention is worse as a dense language model, learned hard eviction changes the useful operating points: sigmoid-gated models delete KV entries with negligible PPL change relative to their own no-eviction references. Under a matched live-cache protocol on the same dense backbones, learned sigmoid gates obtain lower PPL than our H$_2$O and KeyDiff implementations, whereas softmax gates do not uniformly beat these post-hoc methods. The results suggest that attention normalization can substantially affect whether a training-time soft gate transfers cleanly to hard KV deletion.
