Search papers, labs, and topics across Lattice.
The paper introduces Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO$^2$), a hybrid reinforcement learning framework designed to improve exploration in LLM agents by leveraging external memory. EMPO$^2$ combines on-policy and off-policy updates to enhance performance with memory while maintaining robustness without it, addressing the limitation of existing methods in discovering novel states. Experiments on ScienceWorld and WebShop demonstrate that EMPO$^2$ significantly outperforms GRPO, achieving 128.6% and 11.3% improvements, respectively, and exhibits superior out-of-distribution generalization with minimal fine-tuning.
LLM agents can learn to explore novel states and generalize to new tasks with a hybrid on- and off-policy RL framework that leverages memory.
Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO$^2$), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO$^2$ achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO$^2$ demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO$^2$ as a promising framework for building more exploratory and generalizable LLM-based agents.
Microsoft Research
Tsinghua AI