Search papers, labs, and topics across Lattice.
This paper addresses the limitations of Group Relative Policy Optimization (GRPO) in tackling hard reasoning problems by introducing AdaPrefix-GRPO, which dynamically adjusts the length of a correct prefix from a reference solution during training. By maintaining a success rate near 50%, the method optimizes the gradient signal for GRPO, ultimately enhancing model performance on challenging tasks. The results show that AdaPrefix-GRPO significantly improves accuracy鈥攐ver 2x on a 0.6B model鈥攚hile reducing trace length, demonstrating its effectiveness across various model sizes.
Adaptive prefix control can more than double the accuracy of GRPO on hard reasoning tasks while cutting trace length in half.
Group Relative Policy Optimization (GRPO) stalls on a model's hardest problems: when no rollout in a group succeeds, the group-relative advantages vanish and the problem contributes no gradient, wasting the frontier examples we most want to learn from. Prepending a correct prefix of a reference solution raises the success rate, making prefix length a continuous knob on difficulty. Concurrent methods set the knob once; AdaPrefix-GRPO turns it into a feedback controller: throughout training it adjusts how much of the solution each problem gets, holding its success rate near 50%, where GRPO's gradient signal is largest, then withdraws the assistance entirely, so the deployed model solves problems unaided. On hard math, at matched training FLOPs, it more than doubles GRPO's accuracy on held-out problems from the training distribution for a 0.6B model (2.1x), with 1.6x on Qwen3-1.7B and 1.7x on AIME, while roughly halving trace length. The method is implemented in data preparation plus a loss mask on prefix tokens; the trainer is otherwise stock. The smaller the model, the larger the gain.
