Search papers, labs, and topics across Lattice.
This paper introduces a temporal causal evaluation framework for vision-language models (VLMs) that assesses how various information sources influence the generation process during autoregressive decoding. By employing a Structural Causal Model, the authors derive three causal-drive metrics鈥擵isual Causal Drive (VCD), Question Causal Drive (QCD), and Prefix Causal Drive (PCD)鈥攚hich reveal a shift from reliance on visual and question inputs to generated prefixes as decoding progresses. Experimental results demonstrate that these metrics significantly improve the understanding of source-specific generation patterns, with QCD and PCD reducing recovery error by 34.8% and 47.1%, respectively, compared to traditional observational methods.
Causal-drive trajectories reveal a surprising shift in VLM reliance from visual cues to generated prefixes, enhancing our understanding of multimodal generation dynamics.
Vision-language models (VLMs) are increasingly evaluated on complex image and video understanding tasks, yet conventional metrics primarily assess final-answer quality and reveal little about how different information sources shape the generation process. We propose a causal and temporal evaluation framework that traces the evolving roles of visual input, question text, and generated prefixes during autoregressive decoding. Grounded in a Structural Causal Model, we use interventions and backdoor adjustment to derive three step-indexed causal-drive metrics---Visual Causal Drive (VCD), Question Causal Drive (QCD), and Prefix Causal Drive (PCD)---for characterizing source-specific generation patterns without requiring reference answers. Experiments on Qwen3-VL-8B-Instruct across MAVIS, LLaVA-Video-178K, and MiraData, together with cross-model validation on InternVL2-8B, reveal a consistent transition from stronger early question and visual guidance toward increasing reliance on generated prefixes. Randomized-intervention validation shows that QCD and PCD reduce recovery error over observational PMI baselines by 34.8\% and 47.1\%, respectively. On VLMBias, the prefix--visual imbalance score achieves 0.767 AUROC and 0.873 AUPRC for distinguishing prior-driven from visually grounded generations. These results show that causal-drive trajectories provide complementary source-level diagnostics for multimodal generation.
