Search papers, labs, and topics across Lattice.
The authors introduce CLVG-Bench, a new evaluation framework for assessing zero-shot multimodal reasoning in video models via context learning in video generation. This benchmark includes over 1,000 manually annotated metadata across diverse categories like physical simulation and logical reasoning. Experiments using CLVG-Bench reveal that state-of-the-art video models struggle with logically grounded and interactive generation tasks, highlighting multimodal reasoning and physical grounding as key limitations.
Today's best video models achieve near-zero success rates on interactive video generation, revealing a stark gap in multimodal reasoning and physical grounding.
Despite remarkable progress toward general-purpose video models, a critical question remains unanswered: how far are these models from achieving true multimodal reasoning? Existing benchmarks fail to address this question rigorously, as they remain constrained by straightforward task designs and fragmented evaluation metrics that neglect complex multimodal reasoning. To bridge this gap, we introduce CLVG-Bench, an evaluation framework designed to probe video models'zero-shot reasoning capabilities via Context Learning in Video Generation. CLVG-Bench comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories, covering complex scenarios including physical simulation, logical reasoning, and interactive contexts. To enable rigorous and scalable assessment, we further propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations, delivering interpretable textual feedback across diverse video context tasks. Extensive experiments reveal a striking answer to our central question: while state-of-the-art (SOTA) video models, such as Seedance 2.0, demonstrate competence on certain understanding and reasoning subtasks, they fall substantially short with logically grounded and interactive generation tasks (achieving success rates<25% and ~0%, respectively), exposing multimodal reasoning and physical grounding as critical bottlenecks. By systematically quantifying these limitations, the proposed method provides actionable feedbacks and a clear roadmap toward truly robust, general-purpose video models. CLVG-Bench and code are released here.
