Search papers, labs, and topics across Lattice.
This study evaluates the reliance of vision-language models (VLMs) on textual priors by creating a novel benchmark with 540 images and multiple question variants to isolate the effects of phrasing. The results reveal that all tested models, particularly open-weight ones, significantly degrade in performance when faced with questions designed to minimize text leakage, indicating a strong dependency on prior knowledge rather than image content. Notably, post-training adjustments using in-context exemplars and GRPO can effectively reduce this reliance, enhancing model accuracy across various question types.
VLMs exhibit a staggering drop in performance when forced to rely solely on image content, with open models plummeting to just 1-9% accuracy on the most challenging questions.
Vision-language models (VLMs) are increasingly deployed where answers must follow from what is in the image, yet they often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself, which inflates benchmark scores and yields confident but ungrounded answers. Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question. To measure the reliance, we build a 540-image benchmark across six reasoning categories and generate four question variants over the same images, so that phrasing rather than image content is the controlled variable. The hardest variant is written directly from the image to minimize text leakage. We benchmark eleven VLMs spanning small open-weight models to large closed-source systems: every model degrades on the hardest variant, and open models fall furthest. Our central diagnostic is a no-image ablation, which collapses the open-weight models to their text-only floor (1 to 9 percent). Three further analyses, LLM-rated difficulty, low base-to-final textual similarity, and human re-annotation, corroborate genuine image-dependence. In-context exemplars that match how a variant was built recover the most accuracy, and GRPO post-training of a small VLM yields consistent gains across all four variants that transfer to a held-out out-of-distribution set. Textual-prior reliance is measurable and partly trainable away.
