Search papers, labs, and topics across Lattice.
This paper introduces ADAPT, an innovative framework designed to mitigate hallucinations in Multimodal Large Language Models (MLLMs) by directly addressing the degradation of text-to-image cross-attention dynamics during generation. By refining cross-attention visual anchors, implementing an attention-supervised inference mechanism, and utilizing Visual Attention Guidance DPO, ADAPT significantly reduces hallucination rates by 40%-60% across various benchmarks while maintaining multimodal capabilities. The findings underscore the importance of understanding and intervening in attention dynamics to enhance the reliability of MLLMs in generating consistent and contextually accurate content.
Hallucination rates in MLLMs can be slashed by up to 60% through targeted interventions in attention dynamics, revealing a critical pathway to improving model reliability.
Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image. In this paper, we identify an internal signature of hallucination: progressive degradation of text-to-image cross-attention during generation, leading to specific failure patterns like unfocused or biased attention. Existing mitigation strategies are largely outcome-driven and do not explicitly target this failure mode. To address this problem, we propose ADAPT (Attention Dynamics Alignment with Preference Tuning), an attention-based framework that intervenes directly on text-to-image cross-attention dynamics. We propose ADAPT with three key contributions: a cross-attention visual anchor refined from early decoding to provide stable spatial grounding, an attention-supervised inference mechanism that detects and corrects attention drift online, and a Visual Attention Guidance DPO that aligns preferences toward visually grounded responses. Experiments show that each component of ADAPT contributes to hallucination reduction, and the full framework achieves new best results across multiple hallucination benchmarks, reducing hallucination rates by 40%-60% across mainstream backbones while preserving general multimodal capabilities. Our work provides an attention-based perspective on mitigating hallucinations by exploring the model's internal text-to-image cross-attention behaviors. Code is available at https://github.com/yao-ustc/ADAPT
