Search papers, labs, and topics across Lattice.
This study audits the assumption that benchmark accuracy in video large language models (LLMs) reflects true visual understanding by introducing the Visual Dependency Gap (VDG), which quantifies the difference in correctness between original video and black-screen conditions. Through paired McNemar tests on MVBench, the authors demonstrate that while models show significant accuracy differences on original video (p = 0.0003), they perform similarly on black screens (p = 0.53), indicating that visual dependency and accuracy are not inherently linked. The findings reveal that frame diversity is crucial for visual performance, while temporal reasoning contributes minimally, suggesting that current benchmarks may not adequately assess visually grounded capabilities in LLMs.
Accuracy in video LLMs may mask a critical disconnect between visual understanding and benchmark performance, as shown by the newly defined Visual Dependency Gap.
Benchmark accuracy in video large language models (LLMs) is often treated as evidence of visual understanding. We audit this assumption across twenty models spanning 2-78B parameters and ten architecture families. We introduce the Visual Dependency Gap (VDG), the difference in per-question correctness between original-video and black-screen conditions. Paired McNemar tests on MVBench show that accuracy and visual dependency are separable: models differ on original video (p = 0.0003) but not on black screens (p = 0.53). Across models, task-type rankings are stable: Attribute Perception is strongly visual, whereas Temporal Reasoning approaches the language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original video reveals that frame diversity supplies most of the visual benefit, while temporal order contributes near-zero accuracy across sixteen open-weight models. An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. The diagnostic also generalizes to four API-accessed models, whose VDG values range from 0.025 to 0.315. These results motivate VDG as a standard audit for whether video benchmarks measure visually grounded capability. Code is available at https://github.com/JaeLee18/accuracy-without-grounding.
