Search papers, labs, and topics across Lattice.
The paper introduces Retrieval-Augmented Policy Optimization (RAPO), a novel reinforcement learning framework for LLM agents that addresses the limitations of on-policy exploration by incorporating retrieved off-policy step-level traces into the agent's reasoning process. RAPO employs a Hybrid-policy Agentic Rollout strategy to dynamically extend the agent's reasoning receptive field and a Retrieval-aware Policy Optimization mechanism to calibrate policy gradient estimation. Experiments demonstrate that RAPO achieves a 5% average gain across 14 datasets and improves training efficiency by 1.2x compared to existing methods.
LLM agents can explore more effectively by retrieving and reasoning over off-policy step-level traces, leading to significant performance gains and faster training.
Agentic Reinforcement Learning (Agentic RL) has shown remarkable potential in large language model-based (LLM) agents. These works can empower LLM agents to tackle complex tasks via multi-step, tool-integrated reasoning. However, an inherent limitation of existing Agentic RL methods is their reliance on a pure on-policy paradigm for exploration, restricting exploration to the agent's self-generated outputs and preventing the discovery of new reasoning perspectives for further improvement. While recent efforts incorporate auxiliary off-policy signals to enhance exploration, they typically utilize full off-policy trajectories for trajectory-level policy estimation, overlooking the necessity for the fine-grained, step-level exploratory dynamics within agentic rollout. In this paper, we revisit exploration in Agentic RL and propose Retrieval-Augmented Policy Optimization (RAPO), a novel RL framework that introduces retrieval to explicitly expand exploration during training. To achieve this, we decompose the Agentic RL training process into two phases: (i) Hybrid-policy Agentic Rollout, and (ii) Retrieval-aware Policy Optimization. Specifically, we propose a Hybrid-policy Agentic Rollout strategy, which allows the agents to continuously reason over the retrieved off-policy step-level traces. It dynamically extends the reasoning receptive field of agents, enabling broader exploration conditioned on external behaviors. Subsequently, we introduce the Retrieval-aware Policy Optimization mechanism, which calibrates the policy gradient estimation with retrieval reward and importance shaping, stabilizing training and prioritizing retrieval-illuminating exploration. Extensive experiments show that RAPO achieves an +5.0% average gain on fourteen datasets across three agentic reasoning tasks, while delivering 1.2x faster training efficiency.
