Search papers, labs, and topics across Lattice.
MoNe is a lightweight modular neural memory system that enhances long-context inference for frozen pretrained Transformers without the need for retraining. By employing fast-weight neural memory networks with layer-localized gradient updates, it achieves significant efficiency gains, reducing compute and peak GPU memory usage by approximately 80% at 128K tokens while maintaining a minimal parameter overhead of 6.4%. This innovative approach allows for effective context handling beyond the native window size of the backbone model, demonstrating strong performance on challenging benchmarks where traditional in-context learning methods falter.
MoNe slashes compute and memory costs by 80% for long-context inference while enabling Transformers to handle context lengths far beyond their original limits.
We present MoNe, a lightweight modular neural memory that attaches to any frozen pretrained Transformer to enable long-context inference without retraining. MoNe reads context in fixed-size segments via test-time learning of fast-weight neural memory networks with layer-localized gradient updates; at inference, the memory generates keys and values from the query tokens alone, with no context tokens re-read. This two-phase design decouples inference cost from context length, achieving $O(N)$ preprocessing and $O(1)$ query cost with peak GPU memory that does not grow with $N$. At 128K tokens, MoNe reduces both compute and peak GPU memory by approximately 80% compared to ICL with only 6.4% parameter overhead. MoNe generalizes to context lengths far beyond the backbone's native window, achieving strong performance on needle-in-a-haystack and word extraction benchmarks from RULER, where ICL degrades sharply.
