Search papers, labs, and topics across Lattice.
The paper investigates whether visual inputs consistently improve lexical judgments in vision-language models (VLMs) by evaluating their performance on words with varying degrees of concreteness and imagery. They find that real-image contexts often degrade alignment with human ratings, particularly for abstract words where visual evidence is less relevant. Through probing and canonical correlation analysis, the authors attribute this degradation to representational shifts and increased sensitivity to spurious visual cues induced by real images.
Real-world images can actually *harm* a vision-language model's ability to understand abstract concepts, as models latch onto irrelevant visual details.
Visual inputs are often assumed to improve language understanding in multimodal models. We examine this assumption by asking whether vision-language models (VLMs) can distinguish useful visual evidence from incidental image context in lexical judgments. We use human concreteness and imagery ratings because they span words with varying expected visual relevance, from abstract and low-imagery words to concrete and high-imagery words. We find that real-image contexts do not yield consistent gains and often hurt alignment with human ratings, most sharply when visual evidence is least relevant. Through probing and canonical correlation analysis, complemented by an attribution case study, we find that real-image contexts are associated with representational shifts and greater sensitivity to spurious visual cues, coinciding with weaker recoverability of the targeted lexical properties. We further show that instructing models to focus solely on textual content at inference time can reduce this degradation, with the clearest gains on these vulnerable subsets. Our findings suggest that current instruction-tuned VLMs need better calibration of when visual context should inform lexical judgments.
