Search papers, labs, and topics across Lattice.
This paper introduces Gradient Uncertainty-Aware Policy Optimization (GUPO), which enhances the Group Relative Policy Optimization (GRPO) method by addressing the issue of conflicting group gradients in mini-batch updates for post-training Large Language Models (LLMs). By modeling group gradients as random variables and incorporating their uncertainty through a Bayesian approach, GUPO provides a more reliable aggregated update direction, leading to improved policy updates. Experimental results across various benchmarks indicate that GUPO significantly outperforms traditional GRPO methods, particularly in scenarios with high gradient conflict.
GUPO reveals that accounting for gradient uncertainty can dramatically improve policy optimization in post-training LLMs, leading to more effective reasoning capabilities.
Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs) for reasoning. In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update. However, these group gradients can point in conflicting directions. Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates, motivating the need for a reliable aggregated update direction under such conflicts. Standard GRPO aggregation treats the realized group gradients as deterministic contributions and does not account for differences in their reliability during aggregation. To address this issue, we propose Gradient Uncertainty-Aware Policy Optimization (GUPO), which models each group gradient as a random variable under a Bayesian formulation and estimates its probability distribution. GUPO then derives gradient uncertainty using a Dirichlet-based formulation and uses it to calibrate the contribution of each group gradient during aggregation. Extensive experiments on multiple benchmarks demonstrate the effectiveness of GUPO.
