Search papers, labs, and topics across Lattice.
This paper introduces HOLA (Hippocampal Linear Attention), which enhances linear attention models by integrating a bounded exact key-value cache alongside the traditional compressive memory, inspired by the brain's complementary learning systems. The method significantly improves memory retention and retrieval accuracy, achieving a perplexity reduction on Wikitext from 27.32 to 22.92 and outperforming a full-attention Transformer++ model. Additionally, HOLA demonstrates superior performance in needle-in-a-haystack recall tasks, maintaining robustness over longer contexts compared to existing models.
HOLA achieves a remarkable 16.1% reduction in perplexity while retaining critical information that traditional linear attention models typically forget.
Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement. HOLA (Hippocampal Linear Attention) keeps the usual delta-rule state as a compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory: the state models linearly compressible structure, while the cache stores associations that should not be forced through that state. The cache writes without a learned eviction module, keeping tokens with large beta * ||e||, the prediction residual actually committed to the state; a decoupled RMSNorm-gamma cache read then turns these exact KV pairs into sharp retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92 (-16.1%), below a full-attention Transformer++ (26.88), and improves LAMBADA perplexity from 30.95 to 30.26. It also achieves the best linear in-context retrieval and remains much more robust than GDN or a matched HOLA+recency cache on RULER needle-in-a-haystack recall out to 32k tokens (16x its training length).
