Search papers, labs, and topics across Lattice.
This study investigates the mechanisms of post-training reinforcement learning (RL) in enhancing reasoning capabilities in models, specifically through controlled experiments with Qwen-2.5-1.5B. The authors identify two primary processes: strategy selection, activated by diverse supervised fine-tuning (SFT) data, and strategy improvement, facilitated by progressively challenging RL data. These findings not only elucidate the underlying mechanics of RL training but also suggest actionable strategies for improving model reasoning performance.
Understanding how diverse SFT data can activate strategy selection reveals new pathways for enhancing model reasoning capabilities through targeted post-training interventions.
Reinforcement learning has rapidly emerged as a key component in the training of reasoning and coding models, yet it remains poorly understood from a mechanistic perspective. We study how and through what underlying processes capabilities are acquired or enhanced via reinforcement learning post-training. Our analysis, based on controlled math reasoning experiments with Qwen-2.5-1.5B, reveals two core mechanisms: strategy selection and strategy improvement. Our results highlight the role of SFT data and reinforcement learning data in activating these mechanisms, in particular showing how supervising the model on diverse reasoning strategies can enable strategy selection and how increasing difficulty in reinforcement learning data can enable strategy improvement. Taken together, our results provide mechanistic insight into RL training and suggest practical interventions to continue scaling reasoning capabilities.
