Search papers, labs, and topics across Lattice.
This paper reformulates the Proximal Policy Optimization (PPO) algorithm by demonstrating that the gradient of the clipped surrogate can be equivalently represented by a Kullback-Leibler (KL) surrogate with a sample-dependent coefficient. This insight bridges the gap between the two existing PPO forms, revealing that they produce indistinguishable training dynamics across various benchmarks. The findings highlight a structural feature of the clipped surrogate and suggest new avenues for generalizing PPO through the design of the per-sample penalty shape.
The gradient of PPO-Clip can be seamlessly represented by a per-sample KL surrogate, revealing hidden structural insights that could advance reinforcement learning algorithms.
Proximal Policy Optimization (PPO) is the standard policy-gradient algorithm for on-policy reinforcement learning. The literature presents it in two forms, a clipped surrogate that bounds the importance ratio between successive policies and a Kullback-Leibler penalty between them. These forms are treated as separate algorithms with their own gradients, their own hyperparameters, and their own reference implementations, and a sizeable body of empirical work compares them. We show that the gradient of the clipped surrogate is reproduced exactly by a Kullback-Leibler surrogate whose coefficient varies per sample, with closed-form dependence on the importance ratio and the advantage. The identity holds at every minibatch step and across the entire inner loop, and on five MuJoCo continuous-control benchmarks the two losses produce indistinguishable training curves. The reformulation exposes a structural feature of the clipped surrogate that the min notation hides. PPO-Clip's implicit per-sample penalty is a step function at the boundary of the trust region, and the shape of this coefficient is the natural design axis for generalising the algorithm. We sketch the resulting follow-up directions in the discussion.
