Search papers, labs, and topics across Lattice.
This paper evaluates the spatial reasoning capabilities of vision-language models (VLMs) through a benchmark focused on the use of spatial deictic expressions across four languages. The study highlights that VLMs struggle to select demonstratives in a way that aligns with human usage, particularly in relation to the spatial distance of objects. These findings underscore the limitations of current VLMs in understanding context-dependent references, which is crucial for effective communication in multilingual settings.
VLMs misinterpret spatial deictic expressions, failing to match human-like selection based on object proximity.
One of the expected abilities of vision-language models (VLMs) is spatial reasoning ability based on a given text and image. To evaluate the spatial reasoning abilities of VLMs, we focus on the use of spatial deictic expressions, which are defined as spatial expressions whose referent is determined by their situational context, such as ``this''and ``that''. To handle spatial deictic expressions, VLMs must jointly reason over language and visual space, grounding context-dependent references in the image's spatial structure. In addition, selecting appropriate spatial deictic expressions across languages requires VLMs to understand the language-specific spatial distinctions encoded by these expressions. In this paper, we develop a benchmark to evaluate the multilingual ability of VLMs to use spatial deictic expressions in four languages. Our experiments using this benchmark reveal that the tested models use demonstratives in a manner different from that of humans, particularly in selecting the appropriate demonstratives based on the distance to the object.
