Search papers, labs, and topics across Lattice.
The paper investigates the correctness of sampling procedures in discrete diffusion language models (dLLMs) by disentangling denoiser approximation error from sampler-induced error. They introduce a novel sampler-centric oracle framework that uses an exact Hidden Markov Model posterior as a denoiser, derived from a ground-truth Markov chain, to isolate and measure the error introduced by the sampler itself. Their findings reveal that few-step discrete diffusion samplers exhibit distributional inaccuracies even with a perfect denoiser, and that standard evaluation metrics do not reliably indicate sampling correctness.
Even with a perfect denoiser, today's fast discrete diffusion samplers still produce incorrect distributions, revealing a fundamental flaw masked by standard evaluation metrics.
Discrete diffusion language models (dLLMs) provide a fast and flexible alternative to autoregressive models (ARMs) via iterative denoising with parallel updates. However, their evaluation is challenging: existing metrics conflate denoiser approximation error with sampler-induced error from the sampling dynamics, a problem that does not arise for ARMs whose autoregressive sampling exactly reflects the learned probability model. We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain, isolating sampler-induced error in a controlled setting. We show that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length. Moreover, improvements in negative log-likelihood, generative perplexity, or MAUVE do not imply correct sampling. Code is available at https://luhantang.github.io/dllm_sampler
