Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors develop SYNCR, a simulation-grounded benchmark and training pipeline built on Habitat, Kubric, and CLEVRER to systematically diagnose and teach eight cross-video reasoning tasks requiring temporal alignment, identity tracking, and 3D scene integration. Evaluating 22 multimodal LLMs shows that frontier models systematically fail at cross-video physical comparison and multi-view aggregation—failures that parameter scaling alone does not resolve. Supervised fine-tuning on SYNCR nearly doubles model performance (Qwen3-VL-8B jumps from 32.6% to 61.6%) and transfers zero-shot to real-world multi-view temporal ordering on Assembly101 and Panoptic with 9.0–20.5 percentage-point gains.
Scaling model size fails to fix fundamental cross-video reasoning deficits in frontier MLLMs, but targeted supervision from physics simulators reliably transfers zero-shot to real-world multi-view video tasks.
Reasoning across videos requires aligning events, matching identities, comparing motion, and integrating partial observations. Evaluating these capabilities and testing how to improve them requires both reliable labels and targeted supervision. We introduce SYNCR, a simulator-grounded framework that connects these two needs through shared task generators. Built on Habitat, Kubric, and CLEVRER, SYNCR derives answers from environment state and provides 4,000 evaluation questions and 15,960 training questions over disjoint videos, spanning eight cross-video reasoning tasks. Visual ablations and human evaluation assess dependence on the supplied evidence and answer recoverability. Evaluation of 22 multimodal large language models reveals persistent difficulties in physical comparison and scene integration that increasing model size does not consistently resolve. Supervised fine-tuning raises Qwen3-VL-8B's average SYNCR accuracy from 32.6% to 61.6%, with gains extending to task configurations and video sources absent from training for those tasks. Transfer to real footage is most consistent for temporal ordering: accuracy improves by 9.0-20.5 percentage points on constructed Assembly101 and Panoptic ordering sets across three checkpoints spanning two model families and two model sizes, with additional gains on existing temporal reasoning benchmarks. These results establish SYNCR as a controlled setting for diagnosing cross-video reasoning failures, testing their learnability, and identifying where synthetic supervision transfers.