Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Sparse Delta Memory (SDM), a novel architecture that enhances the capacity of gated linear RNNs through a sparse addressing scheme, allowing for significantly improved long-context recall without increasing FLOPs. By replacing the dense key-value outer product with sparse memory operations, SDM achieves better performance on in-context learning and long-context retrieval tasks while maintaining the same number of parameters. Additionally, the model's ability to learn an initial state for its memory leads to further enhancements in common-knowledge and reasoning tasks, demonstrating the effectiveness of this approach in scaling linear attention models.
Scaling linear attention models with Sparse Delta Memory leads to significant improvements in long-context recall and reasoning without the computational burden of larger state sizes.
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.
Meta AI