Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a partial information decomposition (PID) framework to dissect the information processing within Large Vision-Language Models (LVLMs), quantifying the redundancy, uniqueness, and synergy of visual and linguistic inputs. Applying this framework to 26 LVLMs across multiple datasets, the authors identify two distinct task regimes (synergy-driven vs. knowledge-driven) and two family-level strategies (fusion-centric vs. language-centric). They further reveal a consistent three-phase pattern in layer-wise processing, highlighting visual instruction tuning as crucial for learning multimodal fusion.
LVLMs aren't all that they seem: some lean heavily on language priors while others genuinely fuse vision and language, revealing surprisingly diverse strategies across model families.
Large vision-language models (LVLMs) achieve impressive performance, yet their internal decision-making processes remain opaque, making it difficult to determine if the success stems from true multimodal fusion or from reliance on unimodal priors. To address this attribution gap, we introduce a novel framework using partial information decomposition (PID) to quantitatively measure the"information spectrum"of LVLMs -- decomposing a model's decision-relevant information into redundant, unique, and synergistic components. By adapting a scalable estimator to modern LVLM outputs, our model-agnostic pipeline profiles 26 LVLMs on four datasets across three dimensions -- breadth (cross-model&cross-task), depth (layer-wise information dynamics), and time (learning dynamics across training). Our analysis reveals two key results: (i) two task regimes (synergy-driven vs. knowledge-driven) and (ii) two stable, contrasting family-level strategies (fusion-centric vs. language-centric). We also uncover a consistent three-phase pattern in layer-wise processing and identify visual instruction tuning as the key stage where fusion is learned. Together, these contributions provide a quantitative lens beyond accuracy-only evaluation and offer insights for analyzing and designing the next generation of LVLMs. Code and data are available at https://github.com/RiiShin/pid-lvlm-analysis .