Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Addressing the generation incoherence that plagues discrete diffusion language models under aggressive step reduction, the authors show that parallel generation errors stem from cross-entropy fitting conditional token marginals rather than an inherent failure of factorized sampling. They develop AlphaDLM using a sequence-level $\alpha$-loss that interpolates between standard marginal cross-entropy ($\alpha \to 0$) and joint mode-seeking objectives ($\alpha = 1$). Tuning to an intermediate $\alpha$ preserves valid multimodal completions while eliminating inconsistent token combinations, achieving 34.6% on GSM8K in just four forward passes on TinyGSM and showing robust math/code scaling on SDAR-1.7B.
Severe generation degradation in few-step parallel diffusion is an objective mismatch rather than an architectural limitation: swapping cross-entropy for a joint-aware $\alpha$-loss yields coherent generation in as few as four steps.
Discrete diffusion language models can generate multiple tokens in parallel, but reducing the number of denoising steps can lead to inconsistent predictions. Standard cross-entropy training fits conditional token marginals, whereas parallel generation requires consistent joint predictions. We introduce Alpha Diffusion Language Models (AlphaDLM), trained with a sequence-level alpha loss that recovers cross-entropy in the limit of vanishing alpha and has a joint-mode optimum at alpha one. Our analysis characterizes how the objective and factorization jointly determine the fitted distribution. We identify conditions under which intermediate alpha preserves multiple valid completions while excluding invalid token combinations. Trained on TinyGSM, our method achieves 34.6% accuracy on GSM8K with only four model evaluations. We further scale the method to SDAR-1.7B and evaluate it on code and mathematics benchmarks. These results show that changing the training objective can improve the accuracy-computation trade-off of factorized diffusion language models.