Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Random Access Memory Network (RAM-Net), a novel linear attention architecture that addresses the expressivity limitations of fixed-size memory by mapping inputs to high-dimensional sparse vectors that serve as explicit addresses for a large memory state. This design enables exponential state size scaling without increasing the number of parameters, thereby reducing signal interference and improving retrieval fidelity. Experiments show that RAM-Net outperforms state-of-the-art baselines in long-range retrieval tasks and achieves competitive performance in language modeling and zero-shot commonsense reasoning.
Linear attention gets a serious memory upgrade: RAM-Net uses sparse addressing to unlock exponential state scaling without extra parameters, blowing past SOTA in long-range retrieval.
While linear attention architectures offer efficient inference, compressing unbounded history into a fixed-size memory inherently limits expressivity and causes information loss. To address this limitation, we introduce Random Access Memory Network (RAM-Net), a novel architecture designed to bridge the gap between the representational capacity of full attention and the memory efficiency of linear models. The core of RAM-Net maps inputs to high-dimensional sparse vectors serving as explicit addresses, allowing the model to selectively access a massive memory state. This design enables exponential state size scaling without additional parameters, which significantly mitigates signal interference and enhances retrieval fidelity. Moreover, the inherent sparsity ensures exceptional computational efficiency, as state updates are confined to minimal entries. Extensive experiments demonstrate that RAM-Net consistently surpasses state-of-the-art baselines in fine-grained long-range retrieval tasks and achieves competitive performance in standard language modeling and zero-shot commonsense reasoning benchmarks, validating its superior capability to capture complex dependencies with significantly reduced computational overhead.