Search papers, labs, and topics across Lattice.
This paper introduces MindEdit-Bench, a novel benchmark designed to evaluate object-level counterfactual spatial reasoning in vision-language models (VLMs) using real-world indoor scenes. By employing an automatic 3D scene-graph extraction pipeline, the benchmark includes six tasks that assess models' abilities to predict the effects of hypothetical object movements and rotations, revealing significant performance gaps between VLMs and human accuracy. The results show that VLMs achieve only 8%-31% accuracy compared to 81%-97% for humans, highlighting critical areas for improvement in spatial reasoning capabilities of these models.
VLMs struggle with object-level counterfactual reasoning, achieving only a fraction of human accuracy in spatial tasks.
Benchmarks for vision-language models (VLMs) mostly test observational spatial reasoning: models describe relations already visible in the input. Existing what-if tasks typically vary the observer while keeping the scene fixed. Can VLMs instead predict the consequences of hypothetically moving or rotating an object? We introduce MindEdit-Bench, a benchmark of six spatial reasoning tasks built from three-photo smartphone triplets of newly captured indoor scenes via an automatic in-the-wild 3D scene-graph extraction pipeline. Four tasks probe perception and perspective transformation over observed structure; two new tasks, L4 (spatial editing) and L5 (cross-view visibility editing), probe object-level counterfactual reasoning, where correct answers are absent from all input images. Each question provides 8-24 structured answer choices, enabling answer-letter-level diagnosis of spatial and fallback errors. The benchmark covers 120 private indoor scenes not drawn from public datasets, reducing public-data pretraining-overlap risk. Across 15 VLMs on 1,003 human-verified questions, task-wise mean VLM accuracy is only 8%-31%, versus 81%-97% human majority-vote accuracy. The pooled human--best-VLM gap is 53 pp, with at least 39 pp on every task. The structured answer space further reveals non-uniform failures, including weaker camera-depth-axis inference and fallback behavior on difficult visibility-editing cases.
