Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the use of Output Reset (OR) as a smooth one-sided saturation alternative to the clipped surrogate objectives in Proximal Policy Optimization (PPO) and Generalized Relative Policy Optimization (GRPO). The authors find that PPO-OR outperforms PPO-clip by a mean final training-time reward-model score of 0.305, while GRPO-OR exhibits a smaller spread in performance despite not achieving a higher mean score. The results indicate that OR alters optimization behavior, with implications for the stability and performance of policy optimization in large language models.
PPO-OR achieves a significant reward improvement over traditional PPO-clip, highlighting the potential of smooth saturation methods in policy optimization.
PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative. We ask whether Output Reset (OR), a smooth one-sided saturation rule, offers a useful alternative for large language model post-training. PPO-OR and GRPO-OR replace the clipped policy term with an OR squared-margin loss in rollout-relative token log-ratio space; the advantage sign determines the update direction, and a token contributes zero direct OR residual after crossing the favorable margin. We compare PPO-clip with PPO-OR under generalized advantage estimation (GAE), and GRPO with GRPO-OR under group-relative advantages, using \texttt{Llama-3.2-1B-Instruct} on Anthropic \texttt{hh-rlhf} with one shared reward model and three seeds per method. Under GAE, PPO-OR has a mean final training-time reward-model score $0.305$ higher than PPO-clip, with a larger observed across-seed spread. Under group-relative advantages, GRPO-OR does not have a higher mean score, but shows a smaller observed spread, a near-zero terminal OR residual, and a declining overshoot fraction, while the matched GRPO clipped-objective trace remains variable. Both group-relative methods exhibit substantially larger rollout-to-current log-ratio displacement than the GAE methods, and OR does not consistently reduce it. Thus, OR changes optimization behavior in both matched comparisons, but the observed reward effect differs between them. At $G=2$, the GRPO-OR diagnostics do not translate into a reward-score gain. Whether larger groups change this outcome remains open. The reported scores are training-time reward-model measurements, not held-out human-preference performance.