Search papers, labs, and topics across Lattice.
This paper introduces LADBench, a benchmark designed to evaluate logical fault detection in images, featuring over 1,000 synthetic images with logical anomalies across four domains. The study highlights the limitations of current Vision Language Models (VLMs), revealing that even the top-performing models achieve only 70.11% accuracy in identifying these anomalies, indicating significant gaps in their reasoning capabilities. The proposed Tiered Prompting Protocol further illustrates that models struggle to detect logical faults even with explicit guidance, underscoring the need for improved multimodal reasoning frameworks.
Even top-performing Vision Language Models miss 30% of logical anomalies, revealing critical gaps in their reasoning abilities.
Large Vision Language Models (VLMs) excel at visual question answering and semantic grounding, but their capacity for autonomous logical reasoning remains underexplored. Existing anomaly benchmarks emphasize visual errors or direct prompting rather than the physical and social common sense needed for open-world deployment. To address this, we introduce LAD-bench, a benchmark of more than 1,000 curated synthetic images with logical anomalies across four domains: Residential, Urban, Collaborative, and Nature. We further propose a Tiered Prompting Protocol based on progressive disclosure, which measures how much explicit assistance a model needs to localize and reason about a logical fault. Evaluating leading foundation models reveals substantial weaknesses: even the best achieves only 70.11% overall accuracy, showing that implicit logical fault detection remains unsolved. Crucially, models often fail to identify anomalies even after receiving explicit hints in deeper tiers. By surfacing these limitations in sequential multimodal reasoning, LAD-Bench offers a rigorous framework for advancing the safety, reliability, and cognitive alignment of autonomous visual systems. Dataset and Code: https://huggingface.co/datasets/SahasraK/LADBench
