Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Memory Consolidation (MC) as a mechanism to internalize context-induced performance gains in LLMs, inspired by biological memory consolidation, while preserving existing capabilities. A novel five-task-family testbed is proposed to evaluate consolidation and retention under various update regimes. The proposed Cortex method, employing prospective queries and distilling context-conditioned outputs into mergeable FFN updates, achieves state-of-the-art Pareto efficiency on the consolidation-retention frontier across diverse model architectures.
LLMs can permanently "remember" factual edits and skill gains from context, retaining 99.8% of original performance after 50 sequential updates.
Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
Tsinghua AI