Search papers, labs, and topics across Lattice.
The paper introduces Neuromem, a benchmark suite for evaluating external memory modules (EMMs) in large language models (LLMs) under realistic streaming conditions with interleaved insertion and retrieval. Neuromem decomposes the EMM lifecycle into five key dimensions: memory data structure, normalization strategy, consolidation policy, query formulation, and context integration. Experiments using Neuromem on LOCOMO, LONGMEMEVAL, and MEMORYAGENTBENCH datasets reveal performance degradation as memory grows and highlight the importance of memory data structure choice, while showing that compression and generative integration offer limited accuracy gains.
LLM external memory performance degrades as the memory grows across rounds in realistic streaming scenarios, highlighting the need for better memory management strategies.
Most evaluations of External Memory Module assume a static setting: memory is built offline and queried at a fixed state. In practice, memory is streaming: new facts arrive continuously, insertions interleave with retrievals, and the memory state evolves while the model is serving queries. In this regime, accuracy and cost are governed by the full memory lifecycle, which encompasses the ingestion, maintenance, retrieval, and integration of information into generation. We present Neuromem, a scalable testbed that benchmarks External Memory Modules under an interleaved insertion-and-retrieval protocol and decomposes its lifecycle into five dimensions including memory data structure, normalization strategy, consolidation policy, query formulation strategy, and context integration mechanism. Using three representative datasets LOCOMO, LONGMEMEVAL, and MEMORYAGENTBENCH, Neuromem evaluates interchangeable variants within a shared serving stack, reporting token-level F1 and insertion/retrieval latency. Overall, we observe that performance typically degrades as memory grows across rounds, and time-related queries remain the most challenging category. The memory data structure largely determines the attainable quality frontier, while aggressive compression and generative integration mechanisms mostly shift cost between insertion and retrieval with limited accuracy gain.
