Search papers, labs, and topics across Lattice.
This paper presents a unified framework for discrete denoising diffusion models (DDMs), emphasizing the importance of the discrete state space's construction, including tokenization and vocabulary topology. By analyzing various existing formulations as different instantiations within this framework, the authors reveal shared design trade-offs that impact training objectives, inference algorithms, and system optimization. The findings highlight the potential for improved generation capabilities and guide future research directions in the field of discrete data modeling.
A unified framework reveals that the design of the discrete state space fundamentally shapes the performance and capabilities of diffusion models for discrete data.
Discrete denoising diffusion models (DDMs) have recently emerged as a compelling alternative to autoregressive (AR) modeling for discrete data, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are fundamentally shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets. This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.
