Search papers, labs, and topics across Lattice.
This paper analyzes weight changes during one-step distillation of diffusion models, finding that weight direction changes are more significant than weight norm changes. They propose LoRaD, a parameter-efficient adapter using low-rank rotation matrices to model these directional changes. Integrating LoRaD into Variational Score Distillation (VSD) yields WaDi, a one-step distillation framework that achieves state-of-the-art FID scores with significantly fewer trainable parameters and strong generalization capabilities.
One-step diffusion models can achieve state-of-the-art image quality with only 10% of the parameters by focusing on weight direction changes during distillation.
Despite the impressive performance of diffusion models such as Stable Diffusion (SD) in image generation, their slow inference limits practical deployment. Recent works accelerate inference by distilling multi-step diffusion into one-step generators. To better understand the distillation mechanism, we analyze U-Net/DiT weight changes between one-step students and their multi-step teacher counterparts. Our analysis reveals that changes in weight direction significantly exceed those in weight norm, highlighting it as the key factor during distillation. Motivated by this insight, we propose the Low-rank Rotation of weight Direction (LoRaD), a parameter-efficient adapter tailored to one-step diffusion distillation. LoRaD is designed to model these structured directional changes using learnable low-rank rotation matrices. We further integrate LoRaD into Variational Score Distillation (VSD), resulting in Weight Direction-aware Distillation (WaDi)-a novel one-step distillation framework. WaDi achieves state-of-the-art FID scores on COCO 2014 and COCO 2017 while using only approximately 10% of the trainable parameters of the U-Net/DiT. Furthermore, the distilled one-step model demonstrates strong versatility and scalability, generalizing well to various downstream tasks such as controllable generation, relation inversion, and high-resolution synthesis.
