Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Simplified Matching Distillation (SMD), a novel on-policy distillation technique for few-step diffusion models that bypasses the need for auxiliary score models or specialized architectures. By employing a training-only residual proxy called Pivot to capture teacher velocity variations and a repulsion mechanism against weaker teacher predictions, SMD achieves strong performance at 4-NFE. Evaluated across diverse generative tasks and model scales up to 33 billion parameters, SMD outperforms existing few-step baselines and even matches multi-step teachers on several benchmarks.
Few-step diffusion models can achieve state-of-the-art 4-NFE generation with simple on-policy velocity regression and a novel residual proxy, matching multi-step teacher performance without complex auxiliary models.
Few-step distillation reduces the sampling cost of diffusion and flow models, but existing approaches often require auxiliary score models or specialized architectures. We show that simple on-policy velocity regression already provides a strong foundation for 4-NFE generation. To represent teacher velocity variation within each coarse transition, we introduce Pivot, a training-only residual proxy that separates local corrections from the student's interval-mean prediction. We further repel the student from weaker teacher predictions, providing a lightweight mechanism for avoiding undesirable modes. The resulting framework, Simplified Matching Distillation (SMD), remains entirely regression-based, requires neither online score estimation nor adversarial training, and leaves the deployed generator unchanged. We evaluate SMD on image generation and editing, 3D generation, video synthesis, and joint audio--video generation, scaling to 33 billion parameters. At 4-NFE, SMD outperforms strong few-step baselines and surpasses multi-step teachers on several benchmarks. It also supports cross-model, multi-teacher, and causal-video distillation.
Microsoft Research