Search papers, labs, and topics across Lattice.
On-policy distillation (OPD) was investigated for its ability to transfer compositional reasoning skills versus factual knowledge to student language models using a controlled synthetic framework. Results across multiple model families show that reverse-KL OPD reliably transfers compositional skill for unseen reasoning structures but imparts minimal factual knowledge. Further analysis reveals that forward KL promotes factual transfer and student rollouts enhance multi-step reasoning execution, indicating OPD primarily teaches knowledge organization rather than knowledge acquisition.
On-policy distillation excels at teaching LLMs to *use* their existing knowledge for complex reasoning, but doesn't actually teach them *new facts*.
On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning remains unknown. We separate these capabilities using a controlled synthetic framework that measures the student's initial capabilities and independently controls the teacher's additional facts, compositional skill, or both. Across four models from three families, reverse-KL OPD reliably transfers compositional skill across unseen reasoning structures, but transfers minimal factual knowledge. Decoupling the distillation recipe reveals the source of this asymmetry: replacing reverse KL with forward KL restores factual transfer, whereas student rollouts specifically improve the execution of multi-step reasoning. Experiments on recent factual QA and competition mathematics show a similar asymmetry under reverse-KL OPD, yielding notable reasoning gains without factual memory expansion. Together, these results demonstrate that on-policy distillation does not expand a model's parametric knowledge, but instead teaches it to organize and compose the knowledge it already possesses.
