Search papers, labs, and topics across Lattice.
This paper introduces the Attention-Discounted Adaptive Sampler (ADAS), a training-free reranking method designed to enhance parallel decoding in masked diffusion language models by addressing the fragility of token selection. By implementing a soft marginal penalty that discounts candidates based on their interaction with already selected tokens, ADAS improves the quality of predictions without altering the base sampler's stopping rule. Experimental results demonstrate that integrating ADAS with existing samplers like Top-k and Fast-dLLM yields significant improvements in low-NFE performance, achieving average gains of 9.11 and 10.46 percentage points, respectively, with minimal runtime overhead.
Soft attention-discounted reranking can boost masked diffusion model performance by over 10 percentage points while maintaining efficient decoding.
Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled. Existing training-free samplers such as Top-\(k\), Fast-dLLM, and EB-Sampler mainly control how many tokens to reveal, while often ranking candidates by token-wise scores that ignore interactions within the selected set. We propose ADAS, a training-free reranking rule for parallel masked diffusion decoding. ADAS leaves the base sampler's stopping rule unchanged and modifies only subset construction: it greedily discounts a candidate when it attends strongly to already selected positions whose predictions remain uncertain. Unlike graph-constrained methods that turn attention into hard compatibility constraints, ADAS keeps attention continuous and uses it as a soft marginal penalty. Across LLaDA-8B-Base and Dream-7B-Base on GSM8K, MATH500, HumanEval, and MBPP, plugging ADAS into Top-\(k\), Fast-dLLM, and EB-Sampler improves low-NFE performance at matched denoiser evaluations by \(9.11\) and \(10.46\) percentage points on average, respectively, with \(3.1\%\) per-forward runtime overhead. These results show that soft attention-discounted reranking is a simple and modular way to improve quality in highly parallel decoding for masked diffusion language models.
