Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates evidence use in vision-language models (VLMs) by employing monocular metric object-size estimation as an ill-posed diagnostic task, which reveals limitations in how these models utilize visual and language cues. Through counterfactual analysis of 12 open-weight VLMs, the authors find that even the largest models underperform compared to a text-only LLM, particularly in their reliance on target identity and local context, while largely neglecting global scene geometry. The findings highlight critical gaps in evidence selection strategies, suggesting that while VLMs can learn the task, they struggle to effectively incorporate spatial information.
Even the most advanced vision-language models fail to leverage global scene geometry, relying heavily on target identity instead.
Counterfactual analysis is widely used to study evidence use in vision-language models, but its diagnostic value is limited on well-posed tasks: when several cues independently support the same answer, removing one may not change the prediction. We propose monocular metric object-size estimation as an ill-posed diagnostic setting for evidence selection: because physical size cannot be determined from a single uncalibrated image, models must rely on imperfect cues category priors, target appearance, local context, apparent image size, and scene geometry. We assemble Metric VQA ($10{,}813$ dimension queries from Objectron and $331$ tape-measured in-the-wild scenes) and evaluate $12$ open-weight VLMs ($3$--$397$\,B parameters) with counterfactual analysis decomposing six visual and language evidence channels. Even the largest VLMs tested (Qwen3-VL-235B, Qwen3.5-397B, InternVL3.5-241B) trail a text-only frontier LLM on the in-the-wild split. The diagnostic analysis shows: target identity is the most load-bearing cue, target pixels and local context help only some models, apparent size shifts predictions without a directional readout, and global scene geometry is largely unused. We analyze LoRA fine-tuning as an actionable intervention specific to metric estimation: while the task is learnable, the models do not learn to leverage scene geometry.