Search papers, labs, and topics across Lattice.
This study introduces Distract-Bench, a benchmark designed to evaluate the robustness of Vision-Language Models (VLMs) against semantic visual distractions, which are meaningful but irrelevant cues that can mislead reasoning processes. The authors assess eight leading open-source and two closed-source VLMs, revealing that while these models maintain performance under conventional visual corruptions, they exhibit significantly lower robustness when faced with semantic distractions. The findings indicate that such distractions can infiltrate the reasoning process, leading to incorrect conclusions, thereby highlighting a critical gap in the current evaluation of VLM reliability in real-world scenarios.
Reasoning VLMs falter under semantic distractions, often mistaking irrelevant cues for evidence, which can lead to incorrect answers.
Reasoning Vision-Language Models (VLMs) achieve strong performance on complex multimodal tasks, but reliable real-world application requires handling visual inputs that are messier than clean, curated benchmarks. Existing works mainly evaluate such reliability of VLMs through input corruptions, such as noise, blur and weather effects, which make visual evidence harder to perceive. This leaves a critical reliability failure mode underexplored: a model may perceive the evidence correctly, yet reason from plausible but irrelevant and distracting evidence and propagate this mistake to its final answer. To address this gap, we introduce \textbf{Distract-Bench}, a benchmark for evaluating VLM robustness to \textbf{semantic visual distractions}, defined as meaningful but task-irrelevant visual cues added to inputs while preserving the ground-truth answer. We comprehensively evaluate eight leading open-source and two closed-source VLMs across conventional vision corruptions and Distract-Bench. Our results show that Distract-Bench exposes a robustness failure distinct from vision corruptions: reasoning VLMs largely track their non-reasoning base models under perceptual degradation, but show consistently lower robustness to semantic distractions. Further analysis shows that these distractions often enter the reasoning process of VLMs, are treated as evidence, and lead to incorrect answers. Together, these findings reframe robustness evaluation for reasoning VLMs, shifting the focus from degraded perception to distractions for reliable real-world visual reasoning. Our data and code are available at https://github.com/Yizheng-Sun/Distract-Bench.
