Search papers, labs, and topics across Lattice.
The paper introduces AMemGym, an interactive environment for on-policy evaluation and optimization of memory management in LLM-based assistants, addressing limitations of existing static benchmarks. AMemGym uses structured data sampling to predefine user profiles, state-dependent questions, and state evolution trajectories, enabling cost-effective generation of high-quality, evaluation-aligned interactions. Experiments using AMemGym reveal performance gaps in existing memory systems like RAG, long-context LLMs, and agentic memory, demonstrating its utility for both assessment and optimization.
Current memory systems like RAG and long-context LLMs stumble in AMemGym's interactive long-horizon conversations, revealing critical performance gaps in maintaining consistent user state.
Long-horizon interactions between users and LLM-based assistants necessitate effective memory management, yet current approaches face challenges in training and evaluation of memory. Existing memory benchmarks rely on static, off-policy data as context, limiting evaluation reliability and scalability. To address these gaps, we introduce AMemGym, an interactive environment enabling on-policy evaluation and optimization for memory-driven personalization. AMemGym employs structured data sampling to predefine user profiles, state-dependent questions, and state evolution trajectories, enabling cost-effective generation of high-quality, evaluation-aligned interactions. LLM-simulated users expose latent states through role-play while maintaining structured state consistency. Comprehensive metrics based on structured data guide both assessment and optimization of assistants. Extensive experiments reveal performance gaps in existing memory systems (e.g., RAG, long-context LLMs, and agentic memory) and corresponding reasons. AMemGym not only enables effective selection among competing approaches but also can potentially drive the self-evolution of memory management strategies. By bridging structured state evolution with free-form interactions, our framework provides a scalable, diagnostically rich environment for advancing memory capabilities in conversational agents.
