Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces a dual-probe methodology and the Abstraction Gap (AG) metric to disentangle linguistic plausibility from faithful causal reasoning in vision-language models (VLMs). They evaluate eight VLMs on a new benchmark, CAGE, finding that most models exhibit a significant abstraction gap, indicating poor causal reasoning despite fluent language generation. Fine-tuning on causal chain annotations largely fails to close this gap, though one model achieves near-zero AG, suggesting architectural and pretraining choices are key.
VLMs can generate fluent causal explanations that are often causally *wrong*, and a new benchmark reveals that most models struggle to bridge this "abstraction gap" even after fine-tuning.
Vision-language models (VLMs) generate fluent causal explanations, but current evaluations cannot distinguish linguistic plausibility from faithful causal reasoning. We introduce a dual-probe methodology that isolates these properties. The Text-Only Probe measures linguistic quality. The Chain-Text Probe requires models to first generate explicit causal chains. The Abstraction Gap (AG) metric quantifies the normalized performance difference. Evaluating eight VLMs on CAGE (Causal Abstraction Gap Evaluation), a benchmark of 49,500 questions across 5,500 images spanning Pearl's causal hierarchy, we find seven models exhibit AG exceeding 0.50 with text scores of 6--8 but chain scores below 2.5. Fine-tuning on 45,000 chain-annotated examples fails to close the gap. However, one model achieves near-zero AG. The capability exists within current VLM architectures and depends on pretraining and architectural choices. CAGE provides a diagnostic tool for assessing faithful causal reasoning in VLMs.