Search papers, labs, and topics across Lattice.
This paper addresses the issue of structure-level skew in reinforcement learning with verifiable rewards (RLVR), where frequently sampled correct solutions dominate the learning signal at the expense of rarer solutions. The authors introduce Cue-GRPO, a method that redistributes credit based on the rarity of solution forms using deterministic Strategy Cues to create local partitions of verified-correct traces. Experimental results show that Cue-GRPO significantly enhances repeated-sampling performance on Qwen2.5-Math-7B and Llama-3.1-8B-Instruct, particularly benefiting from high sampling budgets while maintaining a minimal training overhead of 6% compared to GRPO.
Rarity-aware credit redistribution can drastically improve reinforcement learning performance by ensuring that rare solutions receive the recognition they deserve.
Reinforcement learning with verifiable rewards (RLVR) com- monly optimizes each correct completion as an independent learning signal. In GRPO, this completion-level uniformity creates structure-level skew: recurring correct solution forms accumulate positive coefficient mass in proportion to how often they are sampled, while rare forms receive limited credit. We formalize this behavior as multiplicity-induced structure-level credit concentration and introduce a partition- conditioned rule that redistributes positive advantages accord- ing to cluster rarity. Cue-GRPO instantiates this rule with- out auxiliary-model inference by using deterministic Strategy Cues to construct rollout-local partitions of verified-correct traces. Across Qwen2.5-Math-7B and Llama-3.1-8B-Instruct, Cue-GRPO improves AIME repeated-sampling performance, with the largest gains at high sampling budgets. Credit Re- distribution (CR) under Judge Partitions (JP) further indi- cates that the proposed redistribution mechanism can oper- ate with judge-derived partitions. Cue-GRPO adds only 6% wall-clock training overhead over GRPO. These results sup- port structure-level credit redistribution as a practical design axis for RLVR, with Strategy Cues providing a low-overhead implementation for competition mathematics. Code is avail- able at https://github.com/CzZ12/When-Correct-Solutions- Repeat-Rarity-Aware-Credit-Redistribution-for-GRPO.
