Search papers, labs, and topics across Lattice.
This paper identifies that diffusion multimodal large language models (dMLLMs) tend to generate premature answers with insufficient visual grounding when used with chain-of-thought reasoning. To address this, they propose Position and Step Penalty (PSP) to delay premature answer generation and Visual Reasoning Guidance (VRG) to amplify visual grounding signals. Experiments show that PSP and VRG improve accuracy by up to 7.5% while achieving over 3x speedup compared to using more diffusion steps.
Diffusion models can reason better and faster: a simple penalty and guidance scheme closes the visual grounding gap with autoregressive models and boosts accuracy by 7.5%.
Diffusion large language models (dLLMs) are emerging as promising alternatives to autoregressive (AR) LLMs. Recently, this paradigm has been extended to multimodal tasks, leading to the development of diffusion multimodal large language models (dMLLMs). These models are expected to retain the reasoning capabilities of LLMs while enabling faster inference through parallel generation. However, when combined with Chain-of-Thought (CoT) reasoning, dMLLMs exhibit two critical issues. First, we observe that dMLLMs often generate the final answer token at a very early timestep. This trend indicates that the model determines the answer before sufficient reasoning, leading to degraded reasoning performance. Second, during the initial timesteps, dMLLMs show minimal dependency on visual prompts, exhibiting a fundamentally different pattern of visual information utilization compared to AR vision-language models. In summary, these findings indicate that dMLLMs tend to generate premature final answers without sufficiently grounding on visual inputs. To address these limitations, we propose Position and Step Penalty (PSP) and Visual Reasoning Guidance (VRG). PSP penalizes tokens in later positions during early timesteps, delaying premature answer generation and encouraging progressive reasoning across timesteps. VRG, inspired by classifier-free guidance, amplifies visual grounding signals to enhance the model's alignment with visual evidence. Extensive experiments across various dMLLMs demonstrate that our method achieves up to 7.5% higher accuracy while delivering more than 3x speedup compared to reasoning with four times more diffusion steps.
