Search papers, labs, and topics across Lattice.
This paper explores the use of random walks on graphs as a framework for evaluating parallel sampling strategies in masked diffusion models (MDMs). By training an MDM on random walk samples without revealing the underlying graph structure, the authors establish a method to quantitatively assess the fidelity of sampling strategies. The findings reveal that the effectiveness of parallel unmasking methods varies significantly with graph structure, and a new bisection sampler is introduced, demonstrating improved speed-quality tradeoffs in language generation tasks.
Parallel sampling strategies in masked diffusion models are not one-size-fits-all; their effectiveness hinges on the underlying graph structure.
In this paper, we propose using random walks on graphs as a verifiable sandbox to study different parallel sampling strategies in masked diffusion models (MDMs). We train an MDM on random walk samples from a fixed graph. The graph or the transition kernel is never shown to the model explicitly and plays the role of latent structure in the sequences, albeit one that is controllable and can be used for quantitative evaluation. Thus, this framework enjoys a Sudoku-like validity check: verifying that an output is a valid walk and estimating the Markov kernel from the walks to measure distribution fidelity. Using simple graphs, we theoretically prove that parallel unmasking via widely used scores like lowest entropy is not uniformly better than a random parallel sampler; the performance critically depends on the structure of the underlying graph. We develop a new bisection sampler for random walks, which takes logarithmic steps in the sequence length and is provably exact under perfect training. Experiments on various graph walk tasks show that different parallel samplers are better for different graphs even in practice. Our initial experiments on a pretrained OpenWebText MDM show that the bisection-style samplers improve speed-quality tradeoffs even for language generation. Together, these results position graph random walks as a mechanistic benchmark for diagnosing and designing parallel samplers for masked diffusion models.
