Search papers, labs, and topics across Lattice.
This paper introduces Reward-Swap Policy Optimization (RSPO), a novel method that addresses the challenges of training large language models (LLMs) on long-horizon, multi-turn tasks with sparse outcome rewards. By integrating a reward-swap mechanism, RSPO effectively utilizes dense process rewards while maintaining alignment with true outcome rewards, thus enhancing the diversity of sampled trajectories and improving convergence rates. Experimental results on benchmarks like WebShop and ALFWorld show that RSPO consistently outperforms various reinforcement learning algorithms, including GRPO, PPO, and GiGPO, leading to significant performance gains.
RSPO transforms the training landscape for LLMs by ensuring that dense rewards enhance learning without sacrificing alignment with true outcomes.
Reinforcement learning holds significant potential for training large language models (LLMs) to handle multi-turn interactive tasks. However, in long-horizon, multi-turn tasks characterized by sparse outcome rewards, directly training with outcome rewards often results in slow convergence due to the sparsity of signals and the lack of fine-grained feedback. Furthermore, the model may fail to learn successful trajectories that are not sampled during training, thereby limiting its performance. Conversely, while employing customized dense process rewards provides richer signals and accelerates convergence, these surrogate rewards may exhibit potential misalignment with the ground-truth outcome rewards. This inconsistency can bias the training direction and ultimately degrade the model's final performance. In this work, we propose Reward-Swap Policy Optimization (RSPO), a method designed to leverage the rich information from dense process rewards to facilitate training with outcome rewards. By utilizing a reward-swap mechanism, RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards, thereby elevating the performance ceiling of the model. We conduct extensive experiments on two challenging agent benchmarks, WebShop and ALFWorld. By applying our method to various reinforcement learning algorithms, including GRPO, PPO, and GiGPO, we demonstrate that RSPO achieves consistent performance improvements across different baselines and benchmarks.
