Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Batch Adaptation Policy Optimization (BAPO), an off-policy Reinforcement Learning with Verifiable Rewards (RLVR) framework, to address the limitations of on-policy methods in LLM post-training, specifically experience waste and reward homogeneity. BAPO dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, ensuring a lower bound on policy improvement. Experiments demonstrate that BAPO achieves a 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks, and resolves 40.7% of problems that base models fail on.
Off-policy reinforcement learning can boost LLM reasoning by 12.5% and solve 40% more problems compared to on-policy methods, simply by re-evaluating and reusing historically difficult samples.
Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.