Search papers, labs, and topics across Lattice.
This paper introduces LOCOMO-CONV, a novel conversational memory benchmark that evaluates memory systems in the context of real conversational usage rather than traditional QA-style probing. The study reveals significant retrieval gaps in conversational framing, particularly with implicit and composed queries, and highlights that strong retrieval does not necessarily enhance response quality. Additionally, the authors identify a phenomenon termed "silent grounding," where memory aids contextual understanding without directly referencing the source information, suggesting that reasoning-based memory elaboration could enhance conversational agents' performance.
Conversational agents may be missing critical context, with implicit queries revealing substantial retrieval gaps that traditional benchmarks overlook.
Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage. We introduce LOCOMO-CONV, a conversa- tional memory benchmark derived from Lo- CoMo with four query styles: dialog, implicit, counterfactual, and composed. Across five rep- resentative memory systems, we evaluate both retrieval recall and end-to-end response qual- ity. Our experiments show that conversational framing exposes substantial retrieval gaps over- looked by QA benchmarks, especially on im- plicit and composed queries, which multi-facet query rewriting narrows for raw-turn mem- ory but not abstractive memory. We further find that strong retrieval does not fully trans- late into response quality, and that implicit queries exhibit silent grounding, where mem- ory improves contextual grounding without ex- plicitly surfacing the gold fact. These results point to reasoning-based memory elaboration as a promising direction, and we release aux- iliary supportive_memory annotations captur- ing conversationally useful context beyond the original gold evidence.
