Search papers, labs, and topics across Lattice.
The paper introduces Trajectory Self-Distillation (T3D), a framework to improve the generation quality of few-step Diffusion Language Models (DLLMs) by distilling the model's own generative trajectories. T3D incorporates Direct Discriminative Optimization (DDO), a reverse-KL objective, to encourage mode-seeking behavior in the student model, focusing on high-probability modes of the teacher. Experiments demonstrate that T3D significantly outperforms existing few-step DLLMs, narrowing the performance gap with full-step decoding while maintaining efficiency.
Few-step diffusion language models get a boost from trajectory self-distillation with direct discriminative optimization, narrowing the quality gap with slower, full-step decoding.
Diffusion large language models (DLLMs) have the potential to enable fast text generation by decoding multiple tokens in parallel. However, in practice, their inference efficiency is constrained by the need for many refinement steps, while aggressively reducing the number of steps leads to a substantial degradation in generation quality. To alleviate this, we propose a trajectory self-distillation framework that improves few-step decoding by distilling the model's own generative trajectories. We incorporate Direct Discriminative Optimization (DDO), a reverse-KL objective that promotes mode-seeking distillation and encourages the student to concentrate on high-probability teacher modes. Across benchmarks, our approach consistently outperforms strong few-step baselines and standard training under tight step budgets. Although full-step decoding remains superior, we substantially narrow the gap, establishing a strong foundation towards practical few-step DLLMs. The source code is available at https://github.com/Tyrion58/T3D.
