Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces RBI-Eval, a novel evaluation framework for assessing when memory content should be integrated into responses by memory-augmented conversational agents. By comparing the behavior of four large language models (LLMs) with and without access to sensitive memory across various retrieval settings, the authors found significant behavioral divergence, particularly in how sensitive memories are integrated. The results indicate that while retrieval systems can mitigate exposure, they do not prevent the integration of sensitive content, highlighting the need for memory-aware decision-making in personalized interactions.
Sensitive memory integration in conversational agents can drop by over 80% depending on the model, revealing critical thresholds for safe personalization.
Long-term memory enables language model agents to support personalized interactions, but it remains unclear when available memories warrant integration into responses. Existing memory evaluations emphasize retrieval accuracy and downstream task utility, while overlooking whether retrieved sensitive memory content is warranted in the current turn. We introduce RBI-Eval, a controlled measurement study built around a probe set that compares model behavior with and without access to sensitive memory under identical benign prompts. We evaluate four base LLMs against a matched no-memory reference across four memory-access settings: full-context exposure and three retrieval systems. Our results reveal substantial behavioral divergence. With memory available, the separation score for sensitive-memory integration decreases by 8.9\%--26.6\% relative to the matched no-memory reference for GPT-5.4-mini, but by 51.1\%--82.9\% for Claude-Sonnet-4.6, DeepSeek-V4-Flash, and Qwen3.5-9B. Control experiments on DeepSeek and GPT-5.4-mini show this effect is specific to sensitive content, rather than general personalization. Retrieval systems reduce exposure but do not eliminate integration once sensitive memory reaches the generator. These findings suggest safe personalization requires memory-aware decisions at both retrieval and generation time.