Search papers, labs, and topics across Lattice.
Sci-VBench is a new benchmark designed to evaluate knowledge- and reasoning-intensive video generation in scientific domains, featuring 1,253 expert-annotated examples across four core disciplines. The benchmark emphasizes the need for models to produce temporally rich videos that require scientific reasoning, moving beyond mere visual plausibility. Analysis reveals that while models achieve high perceptual quality, there is significant variability in their ability to ground prompts and ensure scientific correctness, highlighting a gap between proprietary and open-source systems in modeling complex scientific dynamics.
Despite advances in visual realism, models struggle with accurately capturing scientific reasoning and causal dynamics, revealing a critical gap in video generation capabilities.
We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. It contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering. Each example requires models to generate temporally rich videos that demand scientific reasoning and knowledge-grounded synthesis, going beyond surface-level visual plausibility. We further establish a rubric-based evaluation protocol. Our analysis shows that, under this protocol, both non-expert human evaluators and MLLM-as-Judge systems can achieve relatively high agreement with expert judgments, supporting reproducible evaluation at scale. We benchmark 16 frontier proprietary and open-source models and find that, while automatic perceptual-quality scores cluster tightly across systems, performance on Prompt Grounding and Scientific and Causal Correctness varies substantially, with a pronounced proprietary-open-source gap. These findings show that advances in visual realism have not yet translated into reliable modeling of scientific and causal dynamics.
