Search papers, labs, and topics across Lattice.
This paper introduces Proteus, an innovative approach to long-context sequence modeling that employs incremental memory activation to enhance memory efficiency. By imposing an early bottleneck, Proteus effectively compresses historical context and progressively expands memory capacity, which minimizes interference and improves the retention of later context. The method demonstrates significant performance improvements across various state-of-the-art models in language modeling, reasoning, and long-context retrieval tasks, particularly as context length increases.
Static memory models are holding back performance鈥擯roteus reveals that incrementally activating memory can drastically enhance context retention and reduce interference.
The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state. However, most existing memory models expose a static memory throughout the entire sequence. Because early tokens face no compression pressure, they occupy too many degrees of freedom and"pollute"the memory state, leaving little capacity for later context and increasing interference between what is stored and what arrives next. We study a new paradigm of incremental memory activation, where the effective capacity of memory is progressively expanded as the context grows. Imposing an early bottleneck forces the model to compress history more effectively, while unlocking fresh capacity over time reduces interference and improves retention of later context. We instantiate this paradigm in Proteus, a straightforward mechanism that can be incorporated into a broad class of neural memory architectures at no additional cost. We apply Proteus to state-of-the-art models, including SWLA, Comba, Titans, and Hope-Attention, and observe consistent improvements on standard language modeling and reasoning, as well as on long-context retrieval and understanding, with gains that grow at longer context lengths. Overall, our results show that static memory is suboptimal and that scheduling effective capacity is a simple and broadly applicable tool for sequence modeling.
Mila