Search papers, labs, and topics across Lattice.
This paper introduces a context-weighted modification to discrete flow matching that enhances generative modeling by integrating local context information into the continuous-time Markov chain. By reweighting the training signal through a scaled cross-entropy loss function, the authors achieve a reduction in generative perplexity by up to 63% on OpenWebText, while maintaining competitive quality with a semi-autoregressive block diffusion baseline. The findings underscore the importance of local context in improving both sampling and training efficiency in discrete generative models.
Incorporating local context into discrete flow matching can reduce generative perplexity by up to 63%, revolutionizing how we approach generative modeling.
Discrete flow matching provides a flexible framework for generative modeling on discrete structures. However, the standard factorized training objective exposes the model to targets of varying difficulty, mixing well-conditioned, predictable tokens with ambiguous, high-entropy ones. We empirically demonstrate that the uncertainty over the value of each token is closely related to the density of available context in its neighborhood. Motivated by this observation, we propose a simple modification to the underlying continuous-time Markov chain (CTMC) that incorporates local context information. Our context-weighted sampler improves generation quality with negligible computational overhead, while our scaled cross-entropy loss function reweights the training signal from different tokens and reduces generative perplexity by up to 63% on OpenWebText. Moreover, our approach matches a strong semi-autoregressive block diffusion baseline in quality while retaining the ability to perform generation in any order. These results highlight the role of local context as an important factor in discrete generative modeling and show that simple context-aware modifications can significantly improve both sampling and training efficiency.
