Search papers, labs, and topics across Lattice.
This paper introduces Simplax, an innovative augmentation method for discrete diffusion models that enhances the training objective and reverse transitions without altering the original categorical corruption process. By coupling corrupted categorical states with auxiliary simplex-valued variables, Simplax enables a tractable Rao鈥揃lackwellized reverse-bridge objective and a stochastic reverse sampler, significantly improving generative performance. Empirical results demonstrate that Simplax achieves superior perplexity-entropy tradeoffs on OpenWebText and sets new accuracy records for Sudoku generation, particularly in challenging clue density scenarios.
Simplax not only preserves the integrity of categorical diffusion processes but also enhances generative performance, achieving record-breaking results in both text and puzzle generation tasks.
Discrete diffusion models for categorical generation are defined by a corruption kernel, which determines the intermediate state space and the associated reverse prediction problem. We study uniform discrete diffusion and ask whether its training objective and reverse transitions can be enriched without changing the underlying categorical corruption process. We introduce Simplax, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable while preserving the original uniform diffusion process as its categorical marginal. This augmentation yields a tractable Rao--Blackwellized reverse-bridge objective and a corresponding stochastic reverse sampler, while retaining the corrupted categorical state as the denoiser input. Empirically, Simplax improves the generative perplexity--entropy tradeoff on unconditional OpenWebText generation. On Sudoku, a model trained exclusively on $30$-clue puzzles achieves the highest accuracy among the compared methods across all evaluated clue densities, including the minimum uniquely solvable $17$-clue regime, and also achieves the highest validity in unconditional generation.
