Search papers, labs, and topics across Lattice.
This paper critically examines the assumptions behind privileged self-distillation in on-policy reinforcement learning, specifically addressing how token likelihood changes do not inherently equate to credit for intermediate tokens. The authors establish three formal checks to differentiate between effective scoring of actions, the influence of feedback construction, and the reinforcement of training behavior. Their experiments with a 20B model reveal that naive implementations of additive scoring yield near-chance performance, highlighting the necessity for rigorous validation of scoring mechanisms before attributing credit to token likelihoods.
Token likelihood changes can mislead researchers into overestimating the value of intermediate actions in self-distillation, with experiments showing near-chance performance in scoring effectiveness.
Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens. Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information. A token likelihood change, however, is not automatically outcome credit. We separate three questions: whether the score tracks better actions, whether feedback construction changes what is compared, and what behavior the training loss reinforces. We establish these distinctions formally. When a rollout is scored using hindsight feedback written about that same rollout, its content determines both the tokens and the scoring context, creating direct self-dependence. Using feedback from another rollout of the same problem removes this dependence but does not guarantee a useful score. In matched experiments with a 20B model on AIME 2025, the implemented additive score is near chance (AUC=0.505) and slightly favors incorrect traces after length adjustment. In the paired comparison, the outcome-only control records 64.2\%, versus 24.2\%--33.9\% for five token-score variants. The results motivate validating score meaning, feedback construction, and training behavior separately before calling a likelihood signal credit.
