Search papers, labs, and topics across Lattice.
This paper introduces BlockGen, a blockwise sequence model that leverages uniform-state diffusion models (USDMs) and predictor-corrector samplers to enhance sample quality and performance in discrete diffusion tasks. By addressing the limitations of previous comparisons between uniform and masked diffusion models, BlockGen demonstrates that USDMs can outperform masked diffusion models (MDMs) in block-by-block generation, particularly in low-step scenarios. The findings reveal that while MDMs achieve slightly higher accuracy than USDMs on specific datasets, the performance gap diminishes with increased sampling steps, highlighting the nuanced advantages of blockwise modeling strategies.
Uniform-state diffusion models can outperform masked diffusion models in blockwise generation, challenging prior assumptions about their relative effectiveness.
Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector samplers, uniform-state diffusion models (USDMs) produce samples of higher-quality than masked diffusion models (MDMs), and USDMs equal or outperform MDMs in downstream tasks, even though they exhibit greater perplexity. Two issues remain unresolved. First, existing work compares uniform and masked diffusion with un-informed correctors that re-inject noise at random positions, rather than targeting tokens most likely to be wrong. Second, prior work compares full-sequence diffusion models, so we do not know whether the same conclusion holds when tokens are generated block by block. To address these issues, we introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion. BlockGen trains on a mixture of block sizes and its likelihood interpolates between AR and pure diffusion more finely than models with a fixed block size. BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens without an auxiliary verifier. Under ancestral sampling, uniform outperforms masked in the block-by-block setting, especially in the few-step regime. Under ARPC, the gap closes and reverses at high NFE. With block size $16$ on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText. Find our code at https://github.com/jdeschena/blockgen.
