Search papers, labs, and topics across Lattice.
This study critically evaluates on-policy distillation (OPD) in large language models (LLMs) by analyzing its effects on performance through test-time scaling with varying sampling budgets. The findings reveal that while OPD-trained models excel in average performance across different sampling budgets, the advantage in passing specific tasks diminishes as the sampling budget increases, indicating that OPD primarily enhances sampling efficiency rather than expanding reasoning capabilities. Additionally, a solvability analysis shows that OPD can paradoxically render previously solvable problems unsolvable, suggesting that its benefits may be illusory rather than indicative of true capability enhancement.
OPD may improve sampling efficiency, but it risks making previously solvable problems unsolvable, challenging the notion of true capability expansion in LLMs.
On-policy distillation (OPD) has emerged as a promising post-training technique for enhancing LLM reasoning. It is commonly believed to enable the student model to distill knowledge from a stronger teacher model, thereby expanding capabilities beyond the pre-OPD base model. In this study, we examine this view through the lens of test-time scaling by varying the sampling budget K and evaluating performance with pass@K and avg@K. Specifically, across several OPD variants, we observe that OPD-trained models maintain superior avg@K performance across sampling budgets, while the advantage in pass@K gradually shifts to the pre-OPD base models as K increases. These results suggest that OPD primarily improves sampling efficiency rather than consistently expanding the student's reasoning capability boundary. The pass@K dynamics throughout OPD training further reveal a progressive shift toward stronger small-K performance at the expense of the large-K capability boundary. Furthermore, a problem-level solvability analysis using pass@1024 as the criterion reveals an asymmetry: OPD causes more previously solvable problems to become unsolvable than previously unsolvable problems to become solvable. Together, these findings suggest that, from the perspective of capability expansion, OPD behaves more like an"illusory distillation": its apparent gains arise primarily from improved sampling efficiency rather than from acquiring genuinely new reasoning capabilities from the teacher.
