Search papers, labs, and topics across Lattice.
This paper investigates token-level signals in on-policy distillation (OPD), revealing that raw KL divergence between teacher and student is a poor proxy for learnable information. The authors introduce "token teachability," a metric that quantifies the compatibility between teacher and student distributions, and demonstrate its superior ability to predict learning value in a fixed-context diagnostic. Based on this finding, they propose Teachability-Aware OPD (TA-OPD), a selective distillation method that focuses on high-teachability tokens, achieving better performance than full-token OPD and other baselines while retaining only 5% of the tokens.
Forget blindly chasing teacher-student disagreement in on-policy distillation – focusing on *learnable* disagreement, where the teacher nudges the student within its existing possibilities, unlocks surprisingly efficient learning.
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by prioritizing high-entropy or high-disagreement tokens. We revisit this principle and ask: which token-level teacher signals are actually learnable? Using a fixed-context diagnostic that measures same-context teacher-student KL reduction, we show that raw KL disagreement is a coarse proxy for learning value. It conflates learnable disagreement, where the teacher assigns corrective mass to the student's top-K candidates, with incompatible disagreement, where the teacher places mass mostly off the student's current support. We formalize this local compatibility as token teachability and show that it better predicts fixed-context improvement than raw KL alone. Motivated by this finding, we propose Teachability-Aware OPD (TA-OPD), a lightweight token-position selection method that applies OPD loss to high-teachability positions without reward models or verifiers. Across Qwen2.5 and Qwen 3 teacher-student settings, TA-OPD often surpasses full-token OPD with only 5% retained tokens and improves over entropy- and divergence-based baselines. Our results reframe selective OPD as selecting learnable teacher signals rather than merely salient tokens.
