Search papers, labs, and topics across Lattice.
This paper introduces Ratio-Variance Regularized Policy Optimization (R^2VPO), a novel on-policy RL algorithm that replaces heuristic clipping with an explicit constraint on the policy ratio variance to enforce trust regions. By acting as a "soft brake," R^2VPO preserves gradient signals from novel discoveries and reuses stale data more effectively. Evaluations across LLMs and robotic control tasks demonstrate that R^2VPO achieves substantial performance gains, especially on smaller models and in sparse-reward environments, while also improving sample efficiency.
Ditch hard clipping: ratio-variance regularization offers a principled, "soft brake" approach to trust region policy optimization, unlocking substantial gains in sample efficiency and performance, especially for smaller LLMs.
Standard on-policy reinforcement learning relies on heuristic clipping to enforce trust regions, but this mechanism imposes a severe cost by indiscriminately truncating high-return yet high-divergence updates. We demonstrate that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints, eliminating the need for binary hard clipping. By acting as a distributional ``soft brake'', this approach preserves critical gradient signals from novel discoveries while naturally down-weighting and enabling the reuse of stale, off-policy data. We introduce ${\bf R}^2{\bf VPO}$ (Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework. Extensive evaluations across $7$ LLM scales, spanning both fast and slow reasoning paradigms, and $10$ robotic control tasks demonstrate the generality of the proposed approach. R$^2$VPO achieves substantial performance gains on mathematical reasoning benchmarks, with particularly pronounced improvements on smaller models, while significantly improving sample efficiency. Furthermore, it consistently outperforms PPO baselines in continuous control domains, particularly in sparse-reward and dynamic environments. Together, these findings establish ratio-variance regularization as a principled foundation for stable and data-efficient policy optimization.
Tsinghua AI