Search papers, labs, and topics across Lattice.
This paper introduces VIGIL, a novel reinforcement-learning framework designed to mitigate visual laziness in multimodal large language models (MLLMs) by enhancing causal visual grounding. By penalizing instances of "blind confidence" where models over-rely on language priors despite available visual evidence, VIGIL optimizes the mutual information between visual inputs and generated responses. Experimental results show that VIGIL outperforms existing alignment methods on hallucination and reasoning tasks while maintaining text-only performance, achieving state-of-the-art results with significantly less preference data.
MLLMs can significantly reduce hallucinations and improve reasoning by shifting focus from linguistic shortcuts to causal visual grounding through VIGIL.
Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception, enabling joint reasoning over images and text. Despite inheriting strong reasoning capabilities from LLMs, they remain prone to hallucinations that contradict their visual inputs. Mechanistic studies indicate that this weakness stems from visual laziness: MLLMs encode the correct visual evidence internally, but overly rely on strong language priors during response. Existing alignment methods, such as direct preference optimization, primarily optimize outcome-level rewards based on text. This introduces an optimization bias toward linguistic shortcuts, leading to responses that often contradict the visual evidence. To address this, we propose Visual Information Gain In aLignment (VIGIL), a reinforcement-learning (RL) post-training framework that shifts the focus from numerical reward fitting to causal visual grounding. VIGIL introduces a geometric constraint that explicitly maximizes the mutual information between the visual input and the generated response. We achieve this by penalizing "blind confidence" instances where the model remains improperly certain even when textual-visual attention is masked to create a counterfactual blind state. Extensive experiments show that VIGIL consistently outperforms recent alignment methods across hallucination and reasoning benchmarks without compromising text-only capabilities. Our approach matches the full-data performance of state-of-the-art methods using only 25% of the preference data and even demonstrates emergent spatial grounding capabilities without explicit bounding box supervision.
