Search papers, labs, and topics across Lattice.
ThinkPrior addresses cold-start rollout waste in reinforcement learning with verifiable rewards (RLVR) by using an offline pass from an external anchor model to initialize a Beta difficulty prior over prompts before any target-policy rollouts occur. This targets a major inefficiency in group relative policy optimization (GRPO), where uniform sampling wastes roughly 39% of compute on all-correct or all-wrong groups that yield identically zero gradient. Evaluated on Qwen2.5-Math-7B, ThinkPrior more than halves early zero-advantage silent groups and reduces early rollout waste by nearly a fifth without compromising final accuracy, cutting total generated rollouts by 10.6% when composed with DAPO.
Nearly 40% of GRPO rollouts yield zero gradient due to uninformative all-or-nothing groups, but an offline anchor pass can slash this cold-start compute waste before the target policy generates a single token.
In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation. If all rollouts in a group are correct or all are wrong, their group-relative advantages are identically zero; these zero-advantage silent groups provide no reward-advantage gradient, yet uniform sampling spends 39% of a run's rollouts on them. History-based prompt selection must first spend target-policy rollouts to estimate difficulty, creating a cold start with rollout waste; ThinkPrior instead uses an external anchor in one offline pass to construct a zero-rollout difficulty prior before the first target-policy rollout. The verifier-scored anchor pass rate supplies an external-anchor initialization for a Beta posterior; ThinkPrior selects by expected learnability and then updates from training outcomes, changing neither the loss nor the optimizer. On Qwen2.5-Math-7B across sixteen seeds, ThinkPrior more than halves early silent groups and cuts wasted rollouts through step 30 by nearly a fifth, while we detect no difference in final accuracy. On this 250-prompt pool the fixed-budget result is a reallocation rather than a net saving. The measured ThinkPrior+DAPO composition reduces generated rollouts by 10.6% while both arms retain the same 3840-rollout update budget. The prior requires no target-policy rollout before the first selection, but the posterior thereafter uses target-policy outcomes.
