Search papers, labs, and topics across Lattice.
This paper introduces MeMo, a framework that utilizes explicit multi-layer correlation matrix memories (CMMs) to facilitate efficient memorization, retrieval, and forgetting in language models. By implementing a version-aware operation layer, the framework allows for the modification of knowledge without the need for full model retraining, enabling operations such as replace, obsolete, and rollback to be executed as transactions. The key finding is that these version-aware operations can significantly streamline the process of updating model knowledge while maintaining historical context and supporting efficient memory management.
Forgetting and updating knowledge in language models can now be done through efficient version-aware transactions, eliminating the need for complete retraining.
MeMo proposes language models with explicit multi-layer correlation matrix memories (CMMs), where memorization, retrieval, and forgetting are architectural operations. This paper asks how such memories can reduce the need for retraining when knowledge changes. For changes expressible as MeMo memory associations, the model's accessible knowledge can be updated by editing explicit memories rather than retraining the whole model. We propose a version-aware operation layer in which high-level operations such as replace, obsolete, keep-history, rollback, and trace are compiled into MeMo-native primitive calls over sequences and tokens. The key observation is that a version-aware operation is rarely a single MeMo association. It is an ordered transaction of primitive edits, for example forgetting one sequence-token chain, memorizing another, preserving a historical chain, and recording an inverse program. The framework introduces two auxiliary CMMs: a Version CMM (V-CMM) for mapping version transitions to transaction handles, and a Transaction CMM (T-CMM) for storing reusable change contents and inverse programs. It supports both direct sequence-level edits and structured diff-level inputs, and outlines an evaluation route for update success, rollback, traceability, locality, and transaction reuse.
