Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces NAVIRA, a novel inference-time decoding policy for masked diffusion language models that addresses the correction problem associated with token generation. By decoupling the scoring of tokens and the remasking process, NAVIRA allows for more reliable text generation by preventing early local dependency errors from contaminating the context. Experimental results demonstrate that this approach enhances fluency and maintains diversity, achieving superior performance in LLM-judge evaluations compared to existing methods like PRISM.
Remasking policy, rather than just token quality, is the key to reliable text generation in masked diffusion models.
Masked diffusion language models generate text by iteratively unmasking many tokens in parallel, but this speed comes with a correction problem: tokens generated in the same step are predicted from marginal distributions, and early local dependency errors can later contaminate the context. PRISM addresses this by learning token-level quality scores and remasking unreliable tokens, but its inference rule is coupled: the same forward pass both detects low-quality tokens and computes logits for their replacements, so the erroneous tokens still condition regeneration. We propose NAVIRA, an inference-time decoding policy that separates these two operations and samples remasking positions stochastically. A first forward pass scores tokens; selected tokens are masked; a second forward pass regenerates from the cleaned context. Temperature-controlled remasking reduces repeated correction of the same positions and balances fluency against diversity. In controlled experiments with a 170M masked diffusion language model, decoupling improves fluency, while scheduled stochastic remasking preserves entropy and achieves stronger LLM-judge scores under larger forward-pass budgets. These results show that remasking policy, not only the learned quality signal, is central to reliable masked-diffusion text generation.