Search papers, labs, and topics across Lattice.
This paper introduces MemDefrag, a novel framework that addresses performance degradation in latent memory systems of large language models (LLMs) during memory updates. By leveraging a tracing signal from middle transformer layers, MemDefrag effectively conducts memory defragmentation and implements an informativeness-guided forgetting mechanism. Experimental results demonstrate that MemDefrag significantly enhances knowledge retention and long-context performance, outperforming existing methods like MemoryLLM and M+ by a notable margin.
MemDefrag reveals that a simple tracing mechanism can boost knowledge retention in LLMs by over 25% after multiple memory updates.
Latent memory, which stores past knowledge fragments as per-layer hidden states, has emerged as a promising paradigm (e.g., MemoryLLM and M+) for long-term memory in large language models (LLMs). However, the paradigm suffers from significant performance degradation during memory updates, due to positional encoding misalignment and the absence of any tracing mechanism to distinguish target memory fragments from irrelevant ones. To discover such a tracing mechanism, we probe the layer-wise attention density over stored memory fragments, and find that a small set of middle transformer layers consistently concentrates the highest density on the target fragment - exposing an inherent tracing signal. In light of this, we propose MemDefrag, a training-free and model-agnostic framework that (1) uses a middle-layer tracing signal to conduct memory defragmentation (rank, reorder, and filter memories), and (2) applies an informativeness-guided proportional forgetting mechanism once capacity is exceeded. Experiments show that MemDefrag substantially outperforms MemoryLLM and M+ on knowledge retention (e.g., 43.0% vs. 17.4%/17.6% after 50 memory updates) and long-context benchmarks, and generalizes well across various LLMs and latent-memory variants.
