Search papers, labs, and topics across Lattice.
This paper investigates the reliability of large vision-language models (VLMs) when used as evaluators for image-to-text (I2T) and text-to-image (T2I) tasks. The authors introduce targeted perturbations to outputs, degrading quality along dimensions like object hallucination and spatial reasoning, and then assess whether VLM evaluators can reliably detect these errors. They find that current VLM evaluators exhibit significant blind spots, failing to detect perturbed outputs in many cases, particularly fine-grained compositional and spatial errors, and hallucinated content.
VLM evaluators, despite their growing use, can miss over 50% of targeted errors in generated images and text, especially when those errors involve fine-grained details or spatial relationships.
Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T2I) generation tasks. Despite this growing reliance, the reliability of these Evaluator VLMs remains under explored. In this work, we systematically evaluate the reliability of Evaluator VLMs across both I2T and T2I tasks. We introduce targeted perturbations that degrade output quality along key error dimensions, including object hallucinations, spatial reasoning, factual grounding, and visual fidelity. These perturbations test whether Evaluator VLMs can reliably account for these quality degrading errors in their evaluations. Using a comprehensive benchmark of over 4000 perturbed instances spanning 40 perturbation dimensions, we evaluate 4 prominent VLMs using single-answer scoring, pairwise comparison, and reference-guided paradigms. Our findings reveal that current VLM evaluators exhibit substantial blind spots: they often fail to detect perturbed outputs - in some cases exceeding 50%, struggle particularly with fine-grained compositional and spatial errors, and are often insensitive to hallucinated content that contradicts the input image. Pairwise comparison proves more reliable, though failure rates persist. These results highlight the unreliable nature of current Evaluator VLMs and urge caution in their deployment for benchmarking and development decisions. Code and data have been made publicly available.
