Justin Deschenaux

Categorical sampling cripples discrete diffusion by discarding intermediate uncertainty—diffusing directly over the probability simplex fixes this information collapse, enabling an 8-step distilled model to beat 128-step discrete baselines on GSM8K.

Uniform-state diffusion models can outperform masked diffusion models in blockwise generation, challenging prior assumptions about their relative effectiveness.

Uniform-state diffusion models can now achieve state-of-the-art generative performance in language modeling, thanks to a new Predictor-Corrector sampler that breaks the quality plateau of ancestral sampling.

Uniform-state diffusion models, often overlooked in favor of masked diffusion, surprisingly outperform autoregressive and masked diffusion models on GSM8K when scaled to 1.7B parameters, despite worse perplexity.