Search papers, labs, and topics across Lattice.
The paper introduces VGenST-Bench, a new benchmark for evaluating spatio-temporal reasoning in MLLMs using actively synthesized videos generated with a multi-agent pipeline and human quality control. The benchmark features a 3x2x2 video taxonomy varying spatial scale, perspective, and scene dynamics, along with a hierarchical task suite to isolate visual perception from high-level reasoning. Experiments using VGenST-Bench enable fine-grained diagnosis of spatio-temporal understanding in MLLMs by moving away from passive curation.
Forget passively curated videos: VGenST-Bench actively synthesizes diverse, controlled video scenarios to expose the spatio-temporal reasoning gaps in your MLLMs.
Spatio-temporal reasoning is a core capability for Multimodal Large Language Models (MLLMs) operating in the real world. As such, evaluating it precisely has become an essential challenge. However, existing spatio-temporal reasoning benchmark datasets primarily rely on static image sets or passively curated video data, which limits the evaluation of fine-grained reasoning capabilities. In this paper, we introduce VGenST-Bench, a video benchmark that employs generative models to actively synthesize highly controlled and diverse evaluation scenarios. To construct VGenST-Bench, we propose a multi-agent pipeline incorporating a human quality control stage, ensuring the quality of all generated videos and QA pairs. We establish a comprehensive 3x2x2 video taxonomy, encompassing Spatial Scale, Perspective, and Scene Dynamics to span diverse scenarios. Furthermore, we design a hierarchical task suite that decouples low-level visual perception from high-level spatio-temporal reasoning. By shifting the paradigm from passive curation to active synthesis, VGenST-Bench enables fine-grained diagnosis of spatio-temporal understanding in MLLMs.
