Search papers, labs, and topics across Lattice.
This paper introduces Experience-Augmented Policy Optimization (EAPO), a novel approach that enhances Reinforcement Learning with Verifiable Rewards (RLVR) by utilizing a prior RL-optimized policy to inform decision-making during rollouts. By selectively injecting experience at critical decision points and employing an adapted importance sampling scheme, EAPO addresses the inefficiencies of traditional on-policy optimization and fixed reasoning trajectories. Experimental results show that EAPO significantly outperforms existing RLVR methods on multiple benchmarks, demonstrating its effectiveness in improving the reasoning capabilities of large language models (LLMs).
EAPO revolutionizes LLM reasoning by dynamically integrating prior experiences, leading to consistent performance gains over traditional RLVR methods.
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.
