Search papers, labs, and topics across Lattice.
This paper analyzes the entropic behavior of large language model distillation, demonstrating that forward KL divergence inherently inflates student entropy relative to the teacher, a phenomenon verified empirically. Conversely, reverse KL divergence can deflate student entropy, with the transition between these behaviors depending on the divergence choice and training stage. The findings reveal that divergence acts as an implicit entropy regularizer, particularly crucial in self-distillation where it must counteract entropy reduction from privileged information conditioning.
Forward KL divergence in distillation inflates student entropy, while reverse KL can deflate it, offering a new lens on implicit regularization.
Distillation has become a core primitive of large language model training, but its properties are not yet well understood. We take an entropic perspective, studying how the entropy of the student depends on the data and the divergence that define the distillation objective. We prove that forward KL inflates the entropy of the student above that of the teacher. Since cross-entropy training is a special case, this yields an identity that we verify quantitatively in pretraining and supervised finetuning. Other divergences come with no such guarantee: reverse KL deflates entropy until the gap between student and teacher gets too large, and interpolating between the two changes entropy smoothly early in training but abruptly at convergence. The lower entropy of on-policy distillation comes from token-level reverse KL, not from on-policy sampling. The divergence therefore acts as an implicit entropy regularizer, whose role is clearest in self-distillation: as conditioning on privileged information deflates entropy, the divergence hyperparameters that work best are those that compensate for it.
