Search papers, labs, and topics across Lattice.
This paper introduces Hierarchical Memory Mamba (HMM), an enhancement to the recurrent linear attention model Mamba, which addresses the limitations of fixed-capacity recurrent states in long-sequence modeling by incorporating a lightweight working memory. By extracting paragraph-level semantics from sensory memory and compressing it into persistent long-term memory, HMM enables improved cross-task generalization and overcomes the representation bottleneck inherent in traditional RLAs. Evaluations reveal that HMM significantly boosts retrieval success by 34.3–37.1% and reasoning accuracy by 1.6–14.2% compared to existing Mamba models, while maintaining a minimal increase in parameters and training overhead.
HMM achieves a remarkable 34.3% improvement in retrieval success while only adding 2% to the model's parameters.
Recurrent linear attention models (RLAs) such as Mamba offer efficient linear-time sequence modeling as an alternative to Transformers, yet their fixed-capacity recurrent states limit long-sequence modeling. Drawing inspiration from hierarchical human memory, we propose Hierarchical Memory Mamba (HMM) to address this limitation. Building upon a pre-trained Mamba backbone, HMM integrates a lightweight working memory that extracts slow paragraph-level semantics (PLS) from the fast sensory memory embedded in the backbone's hidden states. The PLS is subsequently compressed into persistent long-term memory for task-relevant retrieval. The hierarchical processing of semantic information overcomes the representation bottleneck of RLAs and endows HMM cross-task generalization through parametric learning, which is not observed in other long-context enhanced Mamba variants. Evaluations on Passkey Retrieval and LongBench-E tasks demonstrate that HMM improves retrieval success by 34.3--37.1% and reasoning accuracy by 1.6--14.2% over strong Mamba-based models, while adding only 2% extra parameters and with minimal training overhead.
