Search papers, labs, and topics across Lattice.
This paper introduces Softmax Advantage Group Estimation (SoftmaxGRPO), a novel approach that enhances group-based reinforcement learning by replacing z-score normalization with temperature-scaled softmax advantages to ensure bounded weights across varying prompt difficulties. The authors derive the finite-group population objective for binary rewards and demonstrate that SoftmaxGRPO reallocates gradient budgets more effectively, leading to significant performance improvements over traditional GRPO. Empirical results show that SoftmaxGRPO achieves a 51.8% success rate on DeepMath and boosts a 1.5B instruction-tuned model's performance from 35.0% to 68.0% on Poetry using lightweight text-similarity rewards.
SoftmaxGRPO reallocates learning signals to improve performance on challenging prompts, achieving a 68.0% success rate on Poetry with minimal reward overhead.
Group-based reinforcement learning objectives such as GRPO can allocate learning signal poorly across prompt difficulty: under binary rewards, group normalization induces a divergent weighting on easy prompts. We introduce Softmax Advantage Group Estimation (SoftmaxGRPO), a drop-in alternative that replaces z-score-normalized group advantages with temperature-scaled softmax advantages, keeping weights bounded regardless of prompt difficulty. For binary rewards, we derive the exact finite-group population objective and identify MaxRL as its low-temperature limit. For bounded scalar rewards, we show that the large-group update exactly optimizes a log-moment-generating-function objective, while a universal finite-group scalar objective cannot exist without additional assumptions on the reward distribution. Empirically, SoftmaxGRPO reallocates measured gradient budget away from near-solved prompts and consistently improves over GRPO under identical rewards. It reaches 51.8% on DeepMath with verifiable rewards and improves a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using only lightweight text-similarity rewards.
