Search papers, labs, and topics across Lattice.
This paper introduces FMLM+, a novel framework that enhances Flow Map Language Models by integrating masking-style noise schedules, enabling non-autoregressive generation with improved flexibility and quality. The authors present Posterior Refinement, a strategy that allows the model to self-correct outputs based on global consistency scoring, achieving performance comparable to discrete models while significantly reducing the number of necessary function evaluations (NFEs). Experimental results across various benchmarks demonstrate that FMLM+ with Posterior Refinement offers a superior speed-quality tradeoff compared to existing Masked Diffusion Models and Flow Map Language Models.
Achieving high-fidelity language generation with 32x fewer function evaluations could revolutionize real-time applications of language models.
Non-autoregressive generation offers a powerful paradigm for iterative refinement, allowing models to recursively critique, erase and regenerate arbitrary subsets of tokens. However, existing non-autoregressive models fail to realize this potential. Masked Diffusion Models (MDMs) suffer from factorization error, causing sample quality to collapse when generating multiple tokens simultaneously. Flow Map Language Models (FMLMs) circumvent this bottleneck via joint sequence transport for excellent few-step generation, but sacrifice the inference-time flexibility of MDMs. We introduce FMLM+, a framework that bridges this gap by equipping FMLM with masking-style noise schedules. While generating the full sequence in a single step, FMLM+ simultaneously scores the global consistency of each token a posteriori. We leverage this to introduce Posterior Refinement, a novel inference-time refinement strategy that enables the model to adaptively self-correct its outputs, matching the performance of discrete baselines with 32x fewer NFEs. Across diverse benchmarks, we demonstrate that FMLM+ with Posterior Refinement improves the speed--quality tradeoff over both MDM and FMLM families, providing a scalable foundation for high-fidelity language modeling.
CMU ML