Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper identifies "Visual Attenuation" in MLLMs, where fine-grained visual signals are suppressed by dominant textual tokens, hindering performance on detailed visual tasks. To combat this, they introduce Variational Information Flow (VIF), a framework using a Conditional Variational Autoencoder (CVAE) to model visual saliency as a latent distribution conditioned on the question-answer pair. Experiments across VQA, fine-grained perception, and visual grounding show VIF improves performance by reversing information loss.
MLLMs struggle with fine-grained visual details not because of the input, but because of an intrinsic "Visual Attenuation" problem that can be solved with a CVAE-based information flow manipulation.
While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding, they frequently falter in fine-grained perception tasks that require identifying tiny objects or discerning subtle visual relationships. We attribute this limitation to Visual Attenuation: a phenomenon where sparse fine-grained visual signals are prematurely suppressed or diluted by dominant textual tokens during network propagation, resulting in a "loss of focus" during the deep-level decision-making process. Existing input-centric solutions fail to fundamentally reverse this intrinsic mechanism of information loss. To address this challenge, we propose the Variational Information Flow (VIF) framework. Adopting a probabilistic perspective, VIF leverages a Conditional Variational Autoencoder (CVAE) to model the visual saliency relevant to the question-answer pair as a latent distribution. As a plug-and-play module, VIF can be integrated into existing architectures. Extensive evaluations across diverse benchmarks, covering General VQA, fine-grained perception, and visual grounding, demonstrate that VIF yields competitive improvements over previous methods, validating its effectiveness in enhancing the fine-grained perception of MLLMs.