Search papers, labs, and topics across Lattice.
Video-Oasis is introduced as a diagnostic suite to evaluate video understanding benchmarks, revealing that 54% of samples can be solved without visual or temporal information. Furthermore, SOTA models perform only slightly better than random chance on the remaining challenging samples. The paper then investigates algorithmic design choices that contribute to robust video understanding, offering guidelines for future research and benchmark creation.
Over half of video understanding benchmark samples are solvable without watching the video, and current models barely outperform random guessing on the rest.
The inherent complexity of video understanding makes it difficult to attribute whether performance gains stem from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, the essential criteria that constitute video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the current landscape of video understanding. In this work, we provide Video-Oasis, a sustainable diagnostic suite designed to systematically evaluate existing evaluations and distill spatio-temporal challenges for video understanding. Our analysis reveals two critical findings: (1) 54% of existing benchmark samples are solvable without visual input or temporal context, and (2) on the remaining samples, state-of-the-art models exhibit performance barely exceeding random guessing. To bridge this gap, we investigate which algorithmic design choices contribute to robust video understanding, providing practical guidelines for future research. We hope our work serves as a standard guideline for benchmark construction and the rigorous evaluation of architecture development. Code is available at https://github.com/sejong-rcv/Video-Oasis.
