Search papers, labs, and topics across Lattice.
This paper investigates whether augmenting attention with an exponentially decaying memory module (RAT+) can improve query-aware sparse inference methods like Quest, MoBA, and SnapKV. Experiments on needle-in-a-haystack tasks show that RAT+ consistently improves accuracy over standard attention across various sparse budgets, even when pretraining OLMo2-7B with the memory module. The authors hypothesize and provide evidence that the memory module enhances both key relevance and query expressiveness, leading to the observed improvements.
Exponentially decaying memory can significantly boost the accuracy of existing sparse attention methods, suggesting a simple way to improve long-context LLM efficiency.
Efficient inference is critical for long-context language models, where attention computation and KV-cache access dominate the cost. Recent work RAT+, introduces a recurrence-augmented attention backbone that enables flexible dilated attention at inference time. In this paper, we investigate whether this exponentially decaying memory can also improve existing query-aware sparse inference methods. Using representative methods including Quest, MoBA, and SnapKV, we show that RAT+ consistently improves accuracy over standard attention across sparse budgets on eight needle-in-a-haystack tasks. We validate these gains both on the released checkpoints from the RAT+ paper and on OLMo2-7B, which we continue pretraining with the added memory module for 10B tokens. Finally, we propose two hypotheses explaining why this memory module benefits query-aware sparse inference and design targeted experiments to support them.
