Search papers, labs, and topics across Lattice.
This paper introduces a typology of reasoning gaps (R1 through R5) that characterizes the coherence between charts, images, and captions in scientific communication, derived from a detailed analysis of 79 traumatic brain injury papers. By systematically categorizing how these multimodal elements interact to convey scientific claims, the authors provide a framework for identifying where grounding succeeds or fails in reader interpretation. The study demonstrates that this typology can predict alignment in judgments between domain experts and non-experts regarding vision-language model descriptions, highlighting the role of contextual knowledge in understanding coherence.
A new typology reveals that the coherence of chart-image pairs in scientific communication can significantly impact how experts and non-experts interpret data.
Charts and images appear together throughout scientific publications, yet most computational work does not characterize their coherence. We argue that a chart, its accompanying image, and the caption that links them form a multimodal unit, and that the inferential work required to read it varies systematically. To capture this variation, we develop a typology of reasoning gaps, R1 through R5, that characterizes how chart, image, and text jointly convey a scientific claim, and the interpretive work this demands of the reader. Some pairs restate the same data, while in other pairs, charts are used to quantify a structure the image localizes, project image content onto an external variable, audit an image-based claim, or jointly construct a frame that neither panel can establish alone. The typology is anchored in the grounding theory of communication and was derived bottom-up, with a neuroscience expert, from a corpus of 79 traumatic brain injury papers and 32 chart-image pairs. Crucially, the levels provide a systematic mechanism for identifying where grounding succeeds or breaks down, rather than leaving it to subjective inference. We show this in a study in which a domain expert and three non-experts judge vision-language model (VLM) descriptions of 25 pairs: the level predicts where their judgments align and where they diverge, isolating the points at which contextual knowledge, not the figure, carries coherence. This typology thus offers figure designers a systematic way to balance text against chart-image pairs, bridging the expert-to-non-expert divide in reading a scientific takeaway.
