Search papers, labs, and topics across Lattice.
This paper identifies and addresses a critical flaw in Multimodal Large Language Models (MLLMs), where disproportionate attention is given to semantically uninformative visual tokens, leading to multimodal hallucinations. The authors introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a novel intervention that purifies structural noise and reallocates attention to more informative visual regions without requiring additional training. Evaluations show that SPAR significantly enhances visual grounding while maintaining low computational costs, marking a substantial improvement in MLLM performance.
MLLMs misallocate attention, leading to hallucinations, but SPAR reclaims focus on meaningful visual information with minimal computational cost.
The empirical success of attention mechanism in Multimodal Large Language Models (MLLMs) often obscures its inherent, subtle flaws. Specifically, MLLMs consistently exhibit disproportionate attention toward certain semantically uninformative visual tokens, a phenomenon termed "register" or "Visual Attention Sinks." While existing inference intervention methods attempt to identify these sink tokens and redistribute their attention weights, such approaches typically treat these tokens in isolation and suffer from computational inefficiency. Instead, we reframe this phenomenon as a generalized textual bias exerted over visual features that extends beyond isolated sink tokens. From this perspective, a pervasive structural bias leads to the dilution of the semantic visual signal, precipitating multimodal hallucinations as the model prioritizes linguistic priors over valid visual evidence. To address this limitation, we introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention. SPAR mitigates this generalized textual bias by purifying structural noise and subsequently redistributing the reclaimed attention budget to the most informative visual regions. Comprehensive evaluations across a diverse spectrum of hallucination benchmarks demonstrate that SPAR effectively restores authentic visual grounding with negligible computational overhead.
