Search papers, labs, and topics across Lattice.
This paper investigates the sampling methods for block diffusion language models (BDLMs), revealing their inherent compatibility with left-to-right decoding compared to traditional masked diffusion models (MDMs). The authors introduce Parallel Autoregressive Decoding (PARD), a novel sampling technique that maintains the left-to-right unmasking structure while enabling parallel token commitment. Experimental results demonstrate that PARD not only enhances generation quality over existing parallel samplers but also provides significant speed improvements compared to standard autoregressive decoding with minimal quality trade-offs.
Parallel Autoregressive Decoding (PARD) achieves superior generation quality and speed by leveraging the natural left-to-right structure of block diffusion language models.
Diffusion language models enable flexible arbitrary-order generation, but existing sampling methods are mostly designed for early masked diffusion models (MDMs). In this work, we study sampling for recent block diffusion language models (BDLMs). We show empirically and analytically that these models are naturally more aligned with left-to-right decoding than MDMs. Based on this observation, we propose Parallel Autoregressive Decoding (PARD), a simple training-free sampling method that preserves left-to-right unmasking structure while allowing parallel token commitment. Extensive experiments show that PARD consistently outperforms existing parallel samplers in generation quality, while achieving substantial speedups over pure AR decoding with only a small quality gap.
