Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces REFT, a method to improve rollout diversity in Reinforcement Learning with Verifiable Rewards (RLVR) by diversifying the first token sampled after the reasoning marker. REFT samples uniformly from the policy's top-N candidate first tokens, broadening the exploration of reasoning paths without affecting the correctness signal. Experiments show that REFT improves Pass@K metrics over DAPO and GRPO baselines across various model sizes and difficulty levels, demonstrating the importance of first-token diversification in RLVR.
RLVR agents are surprisingly sensitive to the very first token they generate, and diversifying only that token unlocks significant gains in exploration and reasoning performance.
Reinforcement Learning with Verifiable Rewards (RLVR) trains reasoning models without labeled trajectories, relying on grouped rollouts to expose the policy to alternative reasoning paths and a verifier to score them. Rollout diversity has accordingly emerged as a central bottleneck in RLVR, with most existing methods broadening exploration through temperature, prefix, or rollout-selection adjustments. We identify a structurally distinguished but overlooked position for broadening this diversity: the first token after the reasoning marker. The policy's first-token distribution exhibits a sharply peaked yet correctness-decoupled phenomenon, and this first token position can broaden the regions a rollout group covers without altering the correctness signal. We introduce REFT (Rollout Exploration with First-Token Diversification), a light addition to the RLVR pipeline that samples first tokens uniformly from the policy's own top-$N$ candidates and allocates rollouts evenly, leaving every other component unchanged. Trained on the resulting diversified rollouts, REFT improves aggregate Pass@1, Pass@8, and Pass@64 over DAPO and GRPO baselines across four base models (0.5B-7B) and three difficulty regimes.