Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Video-Oasis, a diagnostic suite designed to systematically audit existing benchmarks for video understanding, revealing that over half of the benchmark samples can be solved without visual input or temporal context. The authors demonstrate that after filtering out these shortcuts, state-of-the-art models only marginally outperform random guessing on the remaining video-native challenges, indicating a significant capability gap in current video understanding systems. The findings underscore the need for more rigorous evaluation criteria in the development of Video-LLMs, providing a foundation for future research in this area.
Over half of existing video understanding benchmarks can be solved without any visual input, exposing a critical flaw in current evaluation methods.
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.