Search papers, labs, and topics across Lattice.
This paper introduces the Bounded Ratio Reinforcement Learning (BRRL) framework, which addresses the disconnect between trust region methods and the heuristic clipped objective used in Proximal Policy Optimization (PPO). By formulating a novel regularized and constrained policy optimization problem, the authors derive an analytical optimal solution that guarantees monotonic performance improvement and connects trust region policy optimization with the Cross-Entropy Method. Empirical results show that their Bounded Policy Optimization (BPO) and Group-relative BPO (GBPO) outperform traditional PPO and GRPO in terms of stability and performance across various environments, including MuJoCo and Atari.
Bridging the gap between trust region methods and PPO, this new framework guarantees performance improvements while outperforming existing algorithms in stability and effectiveness.
Proximal Policy Optimization (PPO) has become the predominant algorithm for on-policy reinforcement learning due to its scalability and empirical robustness across domains. However, there is a significant disconnect between the underlying foundations of trust region methods and the heuristic clipped objective used in PPO. In this paper, we bridge this gap by introducing the Bounded Ratio Reinforcement Learning (BRRL) framework. We formulate a novel regularized and constrained policy optimization problem and derive its analytical optimal solution. We prove that this solution ensures monotonic performance improvement. To handle parameterized policy classes, we develop a policy optimization algorithm called Bounded Policy Optimization (BPO) that minimizes an advantage-weighted divergence between the policy and the analytic optimal solution from BRRL. We further establish a lower bound on the expected performance of the resulting policy in terms of the BPO loss function. Notably, our framework also provides a new theoretical lens to interpret the success of the PPO loss, and connects trust region policy optimization and the Cross-Entropy Method (CEM). We additionally extend BPO to Group-relative BPO (GBPO) for LLM fine-tuning. Empirical evaluations of BPO across MuJoCo, Atari, and complex IsaacLab environments (e.g., Humanoid locomotion), and of GBPO for LLM fine-tuning tasks, demonstrate that BPO and GBPO generally match or outperform PPO and GRPO in stability and final performance.
ETH