Search papers, labs, and topics across Lattice.
This paper introduces Reverse-Turn Policy Optimization (RTPO), a novel approach to stabilize multi-turn reinforcement learning (RL) training by addressing key sources of instability such as context mismatch and credit assignment issues. By structuring multi-turn rollouts as sparse reverse trees and updating policies in reverse temporal order, RTPO ensures that decisions are aligned with their future consequences, leading to improved training stability. Experimental results demonstrate that RTPO outperforms existing methods, achieving a 21.50% improvement over trajectory-level baselines and a 10.76% improvement over turn-level baselines, underscoring its effectiveness for training tool-using agents.
RTPO eliminates critical instability in multi-turn RL training, achieving over 21% performance improvement compared to traditional methods.
Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single-turn settings. Yet multi-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout-training context mismatch, weak turn-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse-turn formulation. We propose Reverse-Turn Policy Optimization (RTPO), which organizes multi-turn rollouts as sparse reverse trees and performs turn-level policy updates in temporal reverse order, aligning each decision with its downstream continuation. RTPO enables causally consistent turn-level credit assignment and on-policy continuation to control asynchronous drift. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn-level formulation, reduces credit bias, and converges to recursive optimality. Experiments on multi-turn agentic RL benchmarks show that RTPO improves upon trajectory- and turn-level baselines by 21.50% and 10.76%, respectively, highlighting its potential to support more stable training for tool-using agents.
