Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of a classification-based training objective for critics in Proximal Policy Optimization (PPO) applied to large language models, specifically in the context of reinforcement learning with verifiable rewards (RLVR). By replacing the traditional mean-squared-error (MSE) regression with a categorical predictor trained via cross-entropy against smoothed HL-Gauss targets, the authors demonstrate significant improvements in critic signal optimization. The proposed HL-Gauss PPO method outperforms strong PPO and DAPO baselines across various reasoning tasks, highlighting its advantages in calibration and variance reduction in advantage estimation.
Categorical value learning can significantly enhance the performance of PPO critics in reinforcement learning, leading to better calibration and lower variance in advantage estimation.
Proximal Policy Optimization (PPO) for large language models typically trains its critic by mean-squared-error (MSE) regression on scalar value targets. Although scalar MSE is statistically valid for estimating the conditional expected return, sparse binary rewards in reinforcement learning with verifiable rewards (RLVR) make critic optimization and calibration especially consequential: small value errors directly distort the scalar advantages used by PPO. We study whether a classification-based training objective can improve this critic signal. HL-Gauss PPO replaces the scalar MSE head with a categorical predictor over a discretized value support, trained by cross-entropy against smoothed HL-Gauss targets. Its output is decoded to a scalar expectation for standard GAE and PPO; the actor update is therefore unchanged and is not distributional. Across mathematical reasoning, tool-augmented math, and Search-R1, and on both Qwen2.5 and Qwen3 backbones, HL-Gauss PPO consistently improves over strong PPO and DAPO baselines. Controls with one-hot, two-hot, and Bernoulli two-bin critics show that neither a larger output head nor binary classification alone explains the gains. On a common collection of reasoning prefixes, HL-Gauss improves Brier score and calibration error and yields more symmetric, lower-variance advantages. These results position categorical value learning as an effective optimization surrogate for PPO critics in RLVR.
