Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to enhance the exploration capabilities of large language models (LLMs) during Reinforcement Learning with Verifiable Rewards (RLVR) by utilizing guidance from weaker models. By incorporating partial reasoning trajectories from these weaker models, the method effectively disrupts over-confidence in the target model, leading to improved reasoning coverage and performance on mathematical benchmarks. The results show a significant performance increase as the complexity of tasks (represented by larger $k$) grows, while also addressing the issue of entropy collapse without the need for additional supervised fine-tuning or complex reward structures.
Weak model guidance can dramatically enhance LLM reasoning coverage in RLVR, outperforming traditional methods as task complexity increases.
Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@$k$ for large $k$. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as $k$ scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.
