Search papers, labs, and topics across Lattice.
This paper introduces SalArt-VQA, a diagnostic benchmark designed to assess the understanding of salient artifacts in AI-generated images by vision-language models (VLMs). The study reveals that while VLMs can achieve high detection recall for artifacts, they often fail to provide accurate contextual understanding, with the best-performing model correctly answering only 53.26% of detailed artifact-related questions. These findings highlight a critical gap between detection accuracy and grounded understanding, emphasizing the need for more nuanced evaluation methods in VLMs.
High artifact detection rates in VLMs mask significant failures in contextual understanding, with top models misidentifying visual cues in over 46% of cases.
Vision-language models (VLMs) are increasingly used to detect whether AI-generated images contain visible artifacts, yet their ability to analyze such artifacts remains poorly understood. A correct image-level decision can still hide important failures: a model may correctly flag an artifact while relying on the wrong visual cue, selecting the wrong region, or describing a defect that the image does not support. To evaluate these behaviors directly, we introduce SalArt-VQA, a diagnostic benchmark for fine-grained SALient ARTifact understanding in AI-generated images. SalArt-VQA contains 950 images and 3,681 human-authored multiple-choice questions spanning artifact images, matched real reference images, and paired generated reference images. Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, while the reference splits test calibration and abstention when the annotated defect is absent. Across 20 VLMs, SalArt-VQA reveals failures that image-level detection accuracy hides: the strongest model reaches 99.37% detection recall on artifact images but answers all four artifact-side questions correctly on only 53.26% of images. Comparing artifact images with artifact-free references reveals a sensitivity-calibration tradeoff: sensitive models often make unsupported artifact claims, while conservative models avoid false alarms largely by missing real artifacts. These results show that high artifact detection accuracy alone does not imply grounded artifact understanding. SalArt-VQA exposes these hidden failure modes and provides a fine-grained evaluation of whether VLM artifact claims are supported by local visual evidence.
Stanford HAI