Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study identifies a critical flaw in vision-language models (VLMs) termed the Visual Insensitivity Gap, where models fail to utilize visual evidence in 40% to 97% of cases across multiple benchmarks. By introducing the Visual Sensitivity Index (VSI), the authors quantify this gap and demonstrate that it is a property of specific samples rather than the models themselves, with consistent rankings across different architectures. The findings reveal that while VLMs can distinguish between perturbed and clean images, their token predictions remain largely unchanged, indicating a significant encoder-LLM gap that impacts model performance on reasoning tasks.
Vision-language models are often blind to visual evidence, with up to 97% of samples showing a striking insensitivity that undermines their effectiveness.
Vision-language models are evaluated by aggregate accuracy on multimodal benchmarks, a practice that implicitly assumes the model uses its visual input. We show this assumption fails on 40%--97% of samples across six VLMs and three perceptual benchmarks: blurring the question-relevant visual region leaves the next-token distribution nearly unchanged. We name this phenomenon the Visual Insensitivity Gap and quantify it with a per-sample Visual Sensitivity Index (VSI). The gap is a property of samples, not of models: VSI ranks correlate across models (grand-mean Spearman rho=+0.40, permutation p<10^-3), so the same samples are flagged insensitive by VLMs sharing no architectural detail beyond a contrastively pretrained vision tower. The mechanism is concrete: on the insensitive samples, a linear probe on each model's own vision tower distinguishes perturbed from clean images at 0.72--0.79 accuracy, yet the model's argmax token changes on only 2%--11% of the same samples, an encoder--LLM gap above 0.65 on every model. Mapping VSI's diagnostic utility cell by cell surfaces a strong regime (multi-choice reasoning on capable VLMs: AUROC=0.85--0.87) and a weak regime (well-calibrated factuality, where softmax confidence already leads). VSI is not a universal best abstention signal; it is a sample-intrinsic indicator of vision-ignoring failure, best used as a conditional ensemble component.