Search papers, labs, and topics across Lattice.
This study investigates the ability of Vision-Language Models (VLMs) to express epistemic restraint when faced with ambiguous visual evidence, revealing that while these models can internally recognize when abstention is necessary, they fail to communicate this effectively. Utilizing a new benchmark called TRAPSBench, which consists of 1,404 physics scenarios designed to test models' decision-making under uncertainty, the authors introduce the Penalized Epistemic Calibration Score (PECS) to evaluate performance. The findings indicate that even the best-performing VLMs achieve a PECS of only 0.292, highlighting a significant gap between the models' perception capabilities and their output expressions, particularly in visual contexts compared to textual ones.
VLMs can recognize when to abstain from making a decision, yet they fail to express this restraint, achieving a mere 0.292 on a new metric designed to measure this capability.
When visual evidence is occluded or chaotic, models should abstain. In this paper, we show that Vision-Language Models (VLMs) can internally distinguish when abstention is required, but fail to express it anyway. We introduce TRAPSBench, a procedurally generated video benchmark of 1,404 matched physics pairs in which a single targeted change renders the outcome undeterminable from the visual evidence. Furthermore, we introduce Penalized Epistemic Calibration Score (PECS), a new robust metric that requires models to both answer correctly when the outcome is knowable, and abstain when the outcome is not. Across 16 VLMs spanning five families, spontaneous restraint is poor: the best PECS is 0.292. The bottleneck is expression, not perception: linear probes decode answerability from hidden states at up to 0.91 AUROC across physics domains; steering a single-layer void direction causally induces or suppresses abstention. Our results replicate across three open-weight families (Qwen, Gemma, LLaVA). The failure is also more pronounced in visual than textual uncertainty: models detect textual impossibility about 4x more readily than missing visual evidence. Closing this representation--output gap likely requires output-stage interventions.
Meta AI