Search papers, labs, and topics across Lattice.
The paper introduces Instruction-Evidence Contrastive Dual-Stream Decoding (IECD2), a novel decoding framework for VLMs that mitigates hallucination by explicitly balancing linguistic informativeness and visual grounding during generation. IECD2 maintains separate instruction-driven and evidence-driven token probability distributions, fusing them adaptively using a KL-based contrastive gate to suppress tokens unsupported by visual evidence. Experiments across multiple vision-language reasoning tasks demonstrate that IECD2 significantly improves accuracy, reasoning performance, and reduces hallucination compared to existing decoding methods.
VLMs hallucinate less when you force them to "think twice" by contrasting language-driven and vision-driven token probabilities at each decoding step.
Vision-Language Models (VLMs) exhibit strong performance in instruction following and open-ended vision-language reasoning, yet they frequently generate fluent outputs that are weakly grounded in visual evidence. Prior works have shown that instruction prompting further worsens this issue by amplifying language priors, especially when the visual signal is uncertain or ambiguous. To address this challenge, we propose a decoding framework that explicitly balances linguistic informativeness and visual faithfulness during generation. Our method, Instruction-Evidence Contrastive Dual-Stream Decoding (IECD2), maintains two parallel probability distributions of tokens at each decoding step: an instruction-driven stream that promotes expressive and informative responses, and an evidence-driven stream that enforces strict grounding in the image. These two streams are adaptively fused using a symmetric KL-based contrast-based gate, which suppresses tokens favored by language priors but unsupported by visual evidence, while preserving them when both distributions agree. We evaluate IECD2 on multiple datasets spanning various generative vision-language reasoning tasks such as captioning and visual question answering, including POPE, MME, VQAv2, AMBER, MS-COCO, and LLaVA-Bench. IECD2 demonstrates consistent improvements in task accuracy and reasoning performance, alongside a substantial reduction in hallucination across all evaluation metrics compared to state-of-the-art decoding approaches.
