Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates on-policy self-distillation (OPSD) using rich world feedback for RL post-training and compares it to GRPO. The key finding is a strong linear correlation between the initial student-teacher performance gap and the final performance improvement in OPSD, holding across different contexts and model families. This predictive law allows for anticipating OPSD outcomes without full training, offering a way to tune and incorporate world feedback effectively.
Predict whether self-distillation will work *before* you even train, thanks to a surprisingly consistent linear relationship between initial performance gap and final improvement.
Moving beyond simple scalar rewards toward richer world feedback is a natural path to more scalable RL post-training. On-policy self-distillation (OPSD) is a promising recent approach that uses arbitrary feedback as learning signal, yet its reliability compared to established methods, such as GRPO, remains unclear. We identify a strikingly consistent linear correlation between the initial student-self-teacher performance gap and the final performance improvement in OPSD. This relationship holds across context types and model families, providing a powerful predictive law for anticipating the outcome of an OPSD configuration without running the full training procedure. Interestingly, we show that this linear predictability holds with model scale, suggesting a potential basis for new empirical scaling laws on larger models with stronger in-context learning capabilities. In essence, our findings show that OPSD performance can be predicted and tuned before training, offering a principled way to incorporate world feedback as a first-class component of the post-training pipeline.