Search papers, labs, and topics across Lattice.
To evaluate conversational assistants beyond basic factual recall, the authors introduce PRAGMA, a benchmark that tests whether models can integrate multi-session histories to provide recommendations, planning, and decision support under shifting user contexts. Unlike traditional static memory benchmarks, PRAGMA challenges systems to track evolving user preferences, resolve conflicting interaction histories, and correct flawed user assumptions. Empirical evaluation reveals that existing long-context LLMs, external memory frameworks, and standard retrieval pipelines fail significantly at both isolating relevant conversational evidence and executing the longitudinal reasoning required for personalized guidance.
Even state-of-the-art long-context models and memory architectures collapse when forced to synthesize evolving user preferences into actionable advice rather than simply regurgitating past conversational facts.
Large language models (LLMs) are increasingly deployed as personalized assistants that interact with users over extended periods of time. As conversations grow longer, relying on full interaction histories becomes increasingly inefficient and unreliable: long contexts introduce substantial computational overhead, making it difficult for models to consistently identify and utilize the most relevant information for the current request. These challenges have motivated memory systems that structure and retrieve user-specific information. In realistic interactions, users often seek practical guidance such as recommendations, planning, and decision support. Unlike factual recall tasks, personalized guidance requires models to integrate information across multiple past conversations and reason about changing user preferences and experiences. However, existing conversational memory evaluations mainly focus on retrieval and factual recall. To study this challenge, we introduce PRAGMA, a benchmark for evaluating personalized guidance in long-term conversations. PRGAMA contains curated longitudinal conversation histories, evidence annotations, and guidance scenarios grounded in evolving user contexts and incorrect user assumptions. Experiments across retrieval systems, memory systems, and long-context models reveal that current systems struggle both to recover the appropriate conversational evidence and to effectively use it for personalized guidance. Our results highlight the need for memory architectures that support robust conversational retrieval and memory-grounded reasoning beyond evidence recall.
