Search papers, labs, and topics across Lattice.
This paper introduces Raven, a novel linear-time sequence model that optimizes long-context recall by employing a sparse memory routing mechanism that selectively updates a fixed set of memory slots. By interpolating between state-space models and sliding-window attention, Raven effectively reduces interference and mitigates the drawbacks of position-based overwriting, leading to significant improvements in long-range content retention. Experimental results demonstrate that Raven not only competes with but often outperforms existing linear-time models across various recall-intensive benchmarks, maintaining effectiveness even at context lengths up to 16 times its training length.
Raven achieves superior long-context recall by intelligently routing memory updates, outperforming traditional models that struggle with interference and eviction.
Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory. State-based linear models, such as state-space models (SSMs) and linear Transformers, write densely, updating the entire state for each newly arrived token, which leads to interference and makes specific past tokens hard to recover. Sliding-window attention (SWA) exhibits the opposite behavior: it writes sparsely by storing explicit token representations, but only within a fixed window, so recall drops once the relevant token is evicted. Interpolating between these models, we introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing. This lets Raven mitigate SWA's position-based overwriting and hard eviction while reducing interference from dense state updates in SSMs, thereby preserving long-range content much more effectively. Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade. It remains effective when extrapolating to context lengths as large as 16x its training length, with similar gains in hybrid architectures.
