Search papers, labs, and topics across Lattice.
This paper introduces H2HMem, a benchmark designed to evaluate the memory capabilities of large language model agents in complex human-human interactions characterized by multimodal information and intricate discourse phenomena. The benchmark encompasses both dyadic and multi-party conversations, assessing agents on memory recall, reasoning, and application across various modalities. Experimental results indicate significant deficiencies in current agents' abilities to construct, retain, and utilize memories, underscoring the need for advancements in memory handling for future LLMs.
Current LLMs struggle to effectively manage memory in multimodal, multi-participant settings, revealing critical gaps in their design.
Large language model agents are increasingly deployed in human-human interaction settings, such as meeting assistants and clinical documentation systems, where they must observe conversations and retain information for downstream queries. Unlike traditional human-assistant settings, these environments are inherently multimodal, involve complex discourse phenomena such as anaphora and deixis, and contain asynchronous or conflicting information from multiple participants. However, existing memory benchmarks largely focus on single-user, text-only interactions, failing to capture these challenges. To address this gap, we introduce H2HMem, a Human-to-Human Multimodal Memory Benchmark for evaluating memory capabilities in complex human-human interactions. H2HMem includes both dyadic and multi-party conversations with multimodal information streams, and evaluates agents along three dimensions: memory recall, reasoning, and application. Experiments with advanced agents reveal substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions, highlighting substantial room for improvement in next-generation LLM agents.
