Search papers, labs, and topics across Lattice.
The paper introduces LaViSA, a benchmark designed to assess Vision and Language Models (VLMs) in their ability to resolve structural ambiguity by utilizing visual context. By evaluating various VLMs against a dataset of ambiguous sentences paired with corresponding images, the study reveals that while these models can partially leverage visual cues, they still face significant challenges with specific types of ambiguity and nuanced semantic distinctions. This work highlights the limitations of current VLMs in effectively addressing structural ambiguity, underscoring the need for further advancements in this area.
VLMs can only partially resolve structural ambiguity using visual cues, revealing significant gaps in their understanding capabilities.
Structural ambiguity arises when a single sentence admits multiple valid interpretations due to its syntactic structure, posing a fundamental challenge for language understanding. Visual scenes serve as useful cues for resolving such ambiguity, and Vision and Language Models (VLMs) need to be capable of deriving possible semantic interpretations from visual scenes. We introduce Language and Vision Structural Ambiguity (LaViSA), a benchmark designed to evaluate the ability of VLMs to resolve structural ambiguity leveraging visual scenes. LaViSA consists of ambiguous sentences, their disambiguated sentences, and corresponding images of these disambiguated sentences across seven ambiguity categories. Using LaViSA, we conduct a comprehensive evaluation of diverse VLMs, including both proprietary and open-source models with varying parameter scales and reasoning capabilities. Experimental results show that although recent VLMs can leverage visual scenes to resolve structural ambiguity to a some extent, they still struggle with certain ambiguity types and visually subtle semantic distinctions, indicating remaining limitations in resolving structural ambiguity using visual scenes.
