Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To prevent information collapse caused by intermediate categorical sampling in discrete diffusion, Simplex Diffusion Models (SDMs) formulate generative trajectories directly over the probability simplex with closed-form reverse transitions. By propagating continuous belief states across denoising steps, the model retains distributional uncertainty throughout generation while remaining trainable via standard cross-entropy loss and a DDIM-like sampler. SDMs achieve 17.0 GenPPL on OpenWebText in 64 steps and solve 32.1% of GSM8K problems when distilled to just 8 steps, substantially outperforming 128-step discrete diffusion baselines (21.4%).
Categorical sampling cripples discrete diffusion by discarding intermediate uncertainty—diffusing directly over the probability simplex fixes this information collapse, enabling an 8-step distilled model to beat 128-step discrete baselines on GSM8K.
Diffusion models have revolutionized generative modeling for continuous data through the gradual refinement of a belief state. This iterative refinement has not yet carried over to discrete diffusion models, which discard uncertainty at intermediate steps through categorical sampling (information collapse). We propose Simplex Diffusion Models (SDMs), a framework that lifts the diffusion process to the probability simplex to represent beliefs over categories. SDMs admit probability paths with closed-form reverse transitions and can be trained with a simple cross-entropy loss. Contrary to earlier proposals such as Dirichlet Flow Matching which requires integrating an ordinary differential equation, we introduce a DDIM-like sampler with a tunable level of stochasticity. Because SDMs operate on samples on the simplex, they can carry uncertainty across denoising steps, which mitigates information collapse. On OpenWebText, SDMs are competitive with strong Discrete Diffusion baselines, achieving $17.0$ GenPPL at $5.46$ unigram entropy in 64 sampling steps, close to real validation data. Even without Self-Conditioning (SC), SDMs outperform masked and uniform diffusion (with SC or predictor-corrector sampling) on code generation (TinyGSM, $T=0.1$; $49.0\%$ vs. $45.8\%$). Distilled down to 8 steps, SDMs solve $32.1\%$ of GSM8K problems, more than distilled Discrete Diffusion models with 128 steps ($21.4\%$).