Search papers, labs, and topics across Lattice.
This paper introduces the Visual Information-Guided Sampler (VIG-Sampler) for diffusion multimodal large language models (dMLLMs), which enhances the decoding process by prioritizing tokens based on their attention to image tokens rather than solely on their frequency in training data. By penalizing candidate tokens with similar image-attention distributions to previously selected tokens, VIG-Sampler increases the information gain of the decoded subset. Experimental results across seven captioning and VQA benchmarks show that VIG-Sampler significantly outperforms the existing Info-Gain Sampler, achieving an average improvement of 19.3 CIDEr points while reducing the number of decoding steps by half.
VIG-Sampler boosts multimodal generation quality by leveraging visual attention, outperforming traditional methods with fewer decoding steps.
Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation. Starting from a fully masked sequence, dMLLMs progressively decode the sequence by unmasking a subset of the remaining masked positions at each step. Since the selected tokens serve as the prediction context for subsequent steps, deciding which tokens to decode is crucial to the quality of the final output. The most common strategy prioritizes tokens based on a certainty measure that tends to favor tokens frequently observed in the training data. Recent approaches instead order tokens according to their influence on subsequent predictions, but do not explicitly account for the input image. We propose the Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens. We further impose a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded subset. Extensive experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler, which outperforms the Info-Gain Sampler by an average of 19.3 CIDEr points across the captioning benchmarks and surpasses it on COCO Caption while using only half as many decoding steps.
