Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce Exploration-Distillation (ExpDis), a framework that decouples exploration from optimization in RLVR by training explorer policies with novelty incentives and distilling their high-quality, correct trajectories into a separate student policy. This approach mitigates the degradation typically associated with aggressive exploration in RLVR, enabling more effective discovery of novel reasoning strategies. ExpDis surpasses existing methods like DAPO in mathematical reasoning benchmarks, demonstrating improved pass@$k$ scaling and more diverse correct solution generation.
Aggressively scaling exploration in RLVR without sacrificing model quality is now possible by distilling novel strategies from dedicated explorer policies.
Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@$k$ scaling, indicating that ExpDis produces models that generate more diverse correct solutions.