Search papers, labs, and topics across Lattice.
This paper investigates the Uniform Information Density (UID) hypothesis in visually grounded settings using multilingual vision-and-language models to estimate surprisal in image-caption and visual storytelling data across multiple languages. The study reveals that grounding language in perception leads to a more uniform distribution of information compared to text-only contexts, supporting a context-sensitive view of UID. Furthermore, the research shows that in visual narratives, grounding in both image and discourse contexts further enhances information uniformity, particularly at the beginning of discourse units.
Grounding language in visual perception and discourse contexts demonstrably smooths information density, challenging text-centric views of the Uniform Information Density hypothesis.
The Uniform Information Density (UID) hypothesis posits that speakers are subject to a communicative pressure to distribute information evenly within utterances, minimising surprisal variance. While this hypothesis has been tested empirically, prior studies are limited exclusively to text-only inputs, abstracting away from the perceptual context in which utterances are produced. In this work, we present the first computational study of UID in visually grounded settings. We estimate surprisal using multilingual vision-and-language models over image-caption data in 30 languages and visual storytelling data in 13 languages, together spanning 11 families. We find that grounding on perception consistently smooths the distribution of information, increasing both global and local uniformity across typologically diverse languages compared to text-only settings. In visual narratives, grounding in both image and discourse contexts has additional effects, with the strongest surprisal reductions occurring at the onset of discourse units. Overall, this study takes a first step towards modelling the temporal dynamics of information flow in ecologically plausible, multimodal language use, and finds that grounded language exhibits greater information uniformity, supporting a context-sensitive formulation of UID.
