Search papers, labs, and topics across Lattice.
This paper critiques the Group Relative Policy Optimization (GRPO) method in reinforcement learning for language models, highlighting its tendency to concentrate on high-probability responses, which diminishes reasoning capacity. The authors introduce ReCo, a reweighting approach that normalizes response contributions and adjusts the token-level importance ratio to favor less-saturated decision points. Experimental results demonstrate that ReCo enhances performance on mathematical reasoning benchmarks, particularly for larger values of k, while maintaining competitive performance with GRPO for smaller k values.
ReCo reweights GRPO to counteract the detrimental effects of response concentration, leading to significant improvements in reasoning performance on challenging benchmarks.
Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models. Recent work shows that GRPO can reduce the base model's reasoning capacity and underperform it in Pass@k when k is large, indicating reduced coverage of reasoning paths. We find that this reduction is associated with GRPO concentrating on responses that the base model already generates with high probability. We trace this concentration to two mechanisms in the GRPO update. At the response level, high-probability responses dominate the group gradient through repeated occurrence. At the token level, GRPO's importance ratio scales gradients, further reinforcing tokens that become more likely under the current policy. We propose ReCo, a reweighting method that addresses both effects. Response contributions are normalized by their expected occurrence within the rollout group, and the token-level importance ratio is replaced with a variance-based ratio that gives larger update scale to non-saturated decision points where alternative token choices remain plausible. Across Qwen2.5-Math-1.5B/7B and Llama-3.1-8B-Instruct on five mathematical reasoning benchmarks, ReCo improves Pass@k for large values of k and is comparable to GRPO for small values of k.
