Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of maintaining usable long-range information in fixed-size memories as context grows, identifying failures in capacity and cross-position binding. The authors introduce Double-Take (DT), a novel memory architecture employing multi-step unpinned updates, cross-position mixing, and bidirectional change-sensitive writes to improve these aspects. DT significantly enhances end-to-end recall, achieving high AUROC scores at extended context lengths and demonstrating zero-shot transfer capabilities.
Bidirectional, change-sensitive memory updates unlock near-perfect recall across megabytes of context, even when trained on much shorter sequences.
Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while self-associative writes cannot form cross-position bindings. We introduce Double-Take (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the $0.10$ chance floor; adding binding raises it to $0.92$. Trained only on $32\,\mathrm{kb}$ genomic contexts, DT reaches $0.927$ AUROC at $131\,\mathrm{kb}$ and $0.856$ near $1\,\mathrm{Mb}$, while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory.