Search papers, labs, and topics across Lattice.
This paper critiques the standard evaluation of knowledge distillation, arguing that it focuses too narrowly on retained task performance and neglects the loss of crucial teacher model capabilities like uncertainty awareness and safety. The authors reframe distillation as a lossy projection and present a taxonomy of "off-metric" losses that are often overlooked but practically significant. They advocate for a "Distillation Loss Statement" to explicitly document what capabilities are preserved, lost, and why remaining losses are acceptable for a given deployment scenario.
Distilling large models into smaller ones can silently sacrifice crucial capabilities like safety and uncertainty awareness, even if headline metrics stay the same.
This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large, often frontier models into deployable systems, yet headline metrics can hide losses in uncertainty, boundary behavior, process reliability, on-policy stability, grounding, privacy, safety, and diversity. We identify the retention assumption behind current evaluation and reframe distillation as a lossy projection of teacher behavior rather than a faithful copy. We then synthesize existing evidence into a taxonomy of off-metric distillation losses, showing that these losses are concrete, recurring, and measurable. To make the position actionable, we propose scenario-specific preservation targets and a Distillation Loss Statement that reports what was preserved, what was lost, and why the remaining losses are acceptable. The goal is not lossless distillation, but accountable distillation.
