Search papers, labs, and topics across Lattice.
This study introduces MissingBench-Verified, a benchmark that assesses the ability of vision-language models (VLMs) to detect missing parts of objects, a challenge exacerbated by real-world knowledge biases and limited training data. The authors evaluate ten leading VLMs and find that they exhibit significant failure rates in recognizing missing components, even when presented with contradictory evidence from external tools. Notably, attempts to mitigate these failures through tool-assisted verification and other strategies yielded minimal improvements, revealing a critical limitation in the current capabilities of VLMs for inspection tasks.
Vision-language models consistently fail to recognize missing object parts, even when external evidence contradicts their visual perceptions, highlighting a fundamental flaw in their design.
Vision Language Models (VLMs) are well known for hallucinating non-existent objects in images. Objects with missing parts present a unique challenge for VLMs, stemming from both real-world knowledge bias and the scarcity of such images in training data. We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario: when vision-language models fail to recognize that an essential component of an object has been removed. Across ten leading models, we observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception. We further ask whether granting models access to image processing tools (e.g., cropping, contrast adjustment) enables autonomous inspection to resolve these failures. We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement, indicating that this failure mode cannot be addressed through current prompting or post-hoc correction techniques. Our findings highlight a fundamental limitation of current VLM for inspection and monitoring tasks and underscore the need for architectural or training-level interventions that enable models to override internal expectations when confronted with contradictory evidence.
