Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of "copying" in Distribution Matching Distillation (DMD) when compressing pretrained diffusion models into few-step generators. The authors reveal that in high-dimensional settings, distilled students tend to replicate the original noise-data pairings of the teacher model, which contradicts the expectation of latent noise remapping. This copying behavior is shown to be an emergent property of the student's limited geometric freedom rather than a consequence of adversarial objectives or teacher memorization, highlighting important implications for model design and training dynamics in high-dimensional spaces.
Distilled models in high dimensions don't just learn; they copy their teacher's noise-data pairings, revealing unexpected constraints in their geometric freedom.
Distribution Matching Distillation (DMD) compresses pretrained diffusion models into efficient few-step generators by aligning their noised distributions across all scales. In principle, such distribution-level supervision remains agnostic to specific noise-data pairings of the teacher; this provides the student the freedom to remap latent noise, a behavior consistently observed in low-dimensional settings. Surprisingly, we find that in high-dimensional settings, distilled students spontaneously reproduce the original noise-data pairings of the teacher, a phenomenon we term copying. We demonstrate that copying is neither a byproduct of adversarial objectives nor a result of teacher memorization. Instead, our evidence suggests that copying is an emergent property arising from the limited geometric freedom of the student model during high-dimensional distillation.
