Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces UtilMem, a benchmark designed to assess the complex capability of memory utilization in conversational agents, which goes beyond simple factual recall to evaluate how well systems integrate implicit and noisy evidence across extended interactions. The authors find that existing benchmarks fail to predict effective memory utilization, as many systems struggle to synthesize relevant information and resist interference from distractors. Their results highlight a critical gap in current memory architectures, suggesting that advancements in long-term conversational memory will necessitate new designs that enhance evidence integration and robustness.
Strong performance on traditional memory benchmarks doesn't guarantee effective memory utilization in conversational agents, revealing a critical gap in AI memory architectures.
Long-term memory is increasingly important for conversational agents, yet existing benchmarks primarily measure memory through pointwise factual recall: whether a system can recover isolated facts or event-level details from prior interactions. Real-world memory use, however, often requires a more demanding capability: integrating distributed, implicit, and noisy evidence across extended interaction histories into coherent, task-oriented outputs. We call this capability memory utilization. Here, we introduce UtilMem, a diagnostic benchmark comprising 1,717 instances across five domains, designed to evaluate four underexplored aspects of memory utilization: reasoning over dense histories, identifying implicitly relevant memories, synthesizing distributed evidence into summaries, analyses, or plans, and resisting interference from semantically similar distractors. Evaluating a diverse set of retrieval-based and memory-augmented systems, we find that strong performance on conventional factual-memory benchmarks does not reliably translate into effective memory utilization. Moreover, retrieval alone is insufficient: even when relevant evidence is successfully recovered, systems frequently fail to integrate information across sessions or to distinguish useful evidence from plausible distractors. These findings expose a substantial gap between accessing stored information and using it effectively, and suggest that progress in long-term conversational memory will require architectures that explicitly support evidence integration and robustness to retrieval interference. Code is available at https://github.com/peijunallin/UtilMem.