Search papers, labs, and topics across Lattice.
This paper introduces Erase-then-Delta Attention (EDA), a novel memory update rule that separates the processes of erasing outdated information and writing new content in recurrent memory models. By allowing targeted erasure at independently chosen addresses before applying delta-style corrections, EDA enhances memory management and improves performance across various model sizes. Experimental results demonstrate that EDA outperforms existing methods in both short and long-context evaluations, highlighting its effectiveness in maintaining relevant information while suppressing stale data.
EDA not only corrects the current memory write but also actively removes outdated information, leading to superior performance in long-context scenarios.
Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active correction is still anchored to that same write address. As a result, stale information stored at a different address cannot be actively removed before new content is written elsewhere. We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write. The key insight is that recurrent memory models should not only correct the current write, but also selectively suppress outdated memory at an independently chosen address. Concretely, our method first applies a targeted erase step along a learned erase direction, and then performs the standard delta-style corrective write along the current write direction. This preserves the corrective behavior of delta-rule updates while expanding their memory-management capacity. Language-model pretraining experiments across dense 2.5B and MoE 25B-A2.8B model families show that EDA performs best in both settings. The gain persists after 80B-token long-context midtraining of the MoE models, where EDA also performs best in long-context evaluations from 4k to 128k contexts. A compact update analysis and memory-state probes suggest why: EDA keeps the delta-rule corrective write intact while allocating an additional cleanup path most strongly when passive decay is weak. These results suggest that recurrent memory models should decide not only what to write, but also what stale information to erase and where.
