Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Distribution Matching Distillation (DMD) to reduce the number of network evaluations (NFEs) required for high-quality generation in continuous diffusion language models. By leveraging the student model's probabilistic token outputs, DMD offers a unified formulation connecting parameterization to gradient estimators, yielding Simplex-DMD and Reinforce-DMD. Simplex-DMD achieves a 49% reduction in NFEs for a given perplexity and entropy on OpenWebText, while Reinforce-DMD pushes the generation frontier at larger budgets.
Diffusion language models can now generate high-quality text with up to 49% fewer network evaluations by matching output distributions.
Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploiting the student's probabilistic token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024 tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated diffusion baseline at matched entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.