Search papers, labs, and topics across Lattice.
This paper formalizes the task of subject-aware distractor localization, addressing the challenge of identifying visual distractors in photographs while preserving essential compositional elements. The authors introduce the SADL benchmark, comprising 1,800 subject-aware cases and 14,617 annotated candidates, to facilitate systematic evaluation of Vision-Language Models (VLMs) on this task. Their findings indicate that while VLMs excel at identifying distractors, they often over-apply exclusion, leading to the suppression of true distractors, highlighting a critical bottleneck in subject-conditioned reasoning for multimodal systems.
VLMs can identify distractors effectively, but they risk removing essential scene elements, revealing a significant flaw in current approaches to visual composition.
Photographs frequently contain \emph{visual distractors} besides foregrounds and backgrounds of the intended subject, competing for attention and weakening composition. While modern editing tools streamline object removal, identifying which objects to remove remains a mostly manual process. Existing saliency models and open-vocabulary detectors operate without subject awareness, failing to adapt to shifting user intent. Furthermore, context-agnostic removal may disrupt the scene's semantic coherence (e.g., keep the person but remove the chair they are sitting on). To address these limitations, we formalize the task of subject-aware distractor localization, which identifies distractors while retaining compositionally essential objects. This paper introduces \textsc{SADL}, the first real-world benchmark for this task, comprising 1,800 subject-aware cases across 1,000 photographs to enable systematic evaluation and facilitate future research. In total, there are 14,617 annotated candidates, including a robust set of 1,938 hard negatives to stress-test exclusion calibration. We evaluate seven proprietary and open-weight Vision-Language Models (VLMs) on a sequential pipeline of distractor classification followed by exclusion filtering, structured around five inclusion factors and three contextual exclusion rules. Our analysis reveals that VLMs are highly capable of identifying distractors, but then over-apply exclusion, which systematically suppresses true distractors at scale. By exposing this critical bottleneck, \textsc{SADL} provides a foundational diagnostic tool to advance subject-conditioned reasoning in multimodal systems.
