Search papers, labs, and topics across Lattice.
This paper introduces Divergence Regularized Policy Optimization (DRPO), a novel approach to reinforcement learning in large language models (LLMs) that replaces the hard masking of token gradients with a smooth advantage-weighted quadratic regularizer. By maintaining the same trust-region geometry as existing methods while allowing for bounded gradient adjustments, DRPO effectively mitigates harmful policy shifts during training. Experimental results demonstrate that DRPO enhances both the stability and efficiency of LLM reinforcement learning across various model scales and architectures.
Smooth gradient adjustments in DRPO prevent harmful policy shifts, leading to more stable and efficient LLM training.
Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream methods such as PPO and GRPO approximate this control with a ratio-clipping mechanism, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO addresses this mismatch by replacing ratio-based clipping with a divergence-based mask, yielding a trust region defined by the sampled token's absolute probability shift. However, DPPO still relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected. To address this, we propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift. DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights that attenuate diverging updates and provide corrective signals beyond the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves the stability and efficiency of LLM RL training.
