Search papers, labs, and topics across Lattice.
This paper addresses the policy lag problem in asynchronous on-policy reinforcement learning, which is exacerbated by distributed training and high update frequencies. The authors identify the sources of policy lag and propose a novel algorithm, total Variation-based Advantage aligned Constrained policy Optimization (TVAC), to mitigate its effects. Empirical results demonstrate that TVAC exhibits improved robustness to policy lag in both classic RL tasks and a large language model math reasoning task.
Overcome policy lag in distributed RL with TV-ACPO, a method that aligns advantage functions and constrains policy updates, leading to more robust and scalable on-policy learning.
Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: \textit{policy lag}, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose \textit{total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym)} as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.
