Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates whether Vision-Language Models (VLMs) can appropriately express information structure (IS) by distinguishing between discourse-old Topics and discourse-new Foci in visually grounded question answering, using Hungarian syntax as a lens. The study compares six VLMs against human responses, revealing that while VLMs exhibit sensitivity to IS, they tend to over-regularize and collapse onto narrow response templates. This contrasts with humans who employ more variable strategies when navigating the interacting pressures of discourse status, grammatical role, and definiteness.
VLMs can grasp information structure, but they're surprisingly rigid compared to humans, suggesting current benchmarks miss critical aspects of natural language understanding.
Vision-language models (VLMs) are increasingly evaluated for whether they identify the right visual content, but little is known about whether they express such content in a discourse-appropriate form. We address this research gap using information structure (IS), testing whether VLMs distinguish discourse-old Topics from discourse-new Foci in visually grounded question answering. We exploit Hungarian, a language in which Topic and Focus map onto dedicated syntactic positions, making IS choices observable in text. Comparing six VLMs with human participants, we find that models produce IS-relevant constructions, but over-regularise this sensitivity. Under the interacting pressures of discourse status, grammatical role (preference for subject Topics) and definiteness (preference for indefinite Foci), humans choose variable strategies for IS realisation. VLMs, by contrast, collapse onto narrow response templates, resembling mode collapse (Kirk et al., 2024). Our findings suggest that VLM evaluation should look beyond content accuracy to how content is packaged for the discourse.