Search papers, labs, and topics across Lattice.
This paper investigates the performance of multimodal large language models (MLLMs) when tasked with reasoning over visual prompts, revealing a significant accuracy drop of 17.8 points on average when questions are embedded in images rather than presented in text. The authors introduce Visualized Task Semantics (VTS) to assess this semantic channel gap and propose prompt-region grounding to align visual question regions with typed semantics, improving VTS accuracy from 58.0 to 66.3 while maintaining original interface performance. This work highlights the distinct capabilities required for reading task-bearing text and effectively grounding it for reasoning, emphasizing the need for improved multimodal interaction methods.
MLLMs struggle with visual prompts, showing a 17.8-point accuracy drop when questions are embedded in images, revealing a critical semantic gap in multimodal reasoning.
Multimodal large language models increasingly reason over screenshots and documents where the task itself may be written in pixels. Yet benchmarks usually place questions in text, leaving it unclear whether models use the same instruction equally well across channels. We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed. Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average. Models often transcribe the visual question correctly yet fail to use it, exposing a semantic channel gap beyond OCR. To reduce this gap, we present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view. At matched training cost, our method raises four-benchmark VTS accuracy from 58.0 to 66.3 while preserving accuracy on the original interface, and requires no OCR or region metadata at inference. Reading task-bearing text and grounding it as an instruction for reasoning are distinct capabilities.
