Search papers, labs, and topics across Lattice.
This paper reveals that three popular methods for training language models鈥擥roup Relative Policy Optimization (GRPO), GRPO Done Right (Dr. GRPO), and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)鈥攁re fundamentally variations of a single mechanism that adjusts the standard deviation of sampled answers. By analyzing the relationship between answer disagreement and training updates, the authors demonstrate that the effectiveness of these methods hinges on the group-standard-deviation identity, where a split group yields the most informative training signals. Their findings, validated on the Big-Math dataset, emphasize that what appears to be a simple normalization step is crucial for directing learning and determining the weight of various problems.
The same underlying principle governs three seemingly distinct training methods, revealing that the disagreement among answers is the key driver of effective learning.
Three of the most popular methods for training language models to reason look like three different tricks. They are not. All three adjust a single number: standard deviation, reflecting how much a prompt's sampled answers disagree. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree. Group Relative Policy Optimization (GRPO) divides by this number, GRPO Done Right (Dr. GRPO) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) discards the groups where it is zero. Each is presented as its own fix, yet this paper proves they are three settings of one dial. That dial is not cosmetic: for right-or-wrong rewards, the disagreement is exactly the size of the training update, the group-standard-deviation identity. A split group teaches the most, while a unanimous group teaches nothing and falls silent. The same result says which problems deserve the most weight and how many tries each one needs. This paper confirms the intuition on a large real difficulty dataset (Big-Math) and in a controlled training run. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly.
