Search papers, labs, and topics across Lattice.
This study investigates the role of attention in diffusion large language models (dLLMs) and reveals that tokens with stronger attention to unmasked context lead to enhanced generation stability and reasoning capabilities. To leverage these insights, the authors introduce AGDO, an attention-guided denoising and optimization framework that aligns training and optimization processes with attention-derived dependencies. Experimental results show that AGDO significantly outperforms existing post-training methods on mathematical and coding benchmarks, highlighting its effectiveness in improving reasoning performance in dLLMs.
Attention-guided denoising can dramatically enhance reasoning performance in diffusion language models, outperforming traditional post-training methods.
Diffusion large language models (dLLMs) offer an efficient alternative to autoregressive models through parallel decoding, yet existing post-training methods largely rely on random masking strategies that overlook intrinsic token dependencies. In this work, we present an empirical analysis of attention in dLLMs and show that tokens attending more strongly to unmasked context exhibit greater generation stability and play a critical role in reasoning. Motivated by these findings, we propose AGDO, an attention-guided denoising and optimization framework that aligns both training and optimization with attention-derived dependencies. AGDO determines the denoising order based on attention structure and emphasizes attention-critical tokens during supervised fine-tuning and reinforcement learning. Experiments on mathematical and coding benchmarks demonstrate that AGDO consistently improves reasoning performance, outperforming state-of-the-art post-training methods for dLLMs.
