Search papers, labs, and topics across Lattice.
The paper introduces VisPhyWorld, a framework for evaluating physical reasoning in MLLMs by requiring them to generate executable simulator code from visual observations, enabling direct inspection and falsification of inferred world representations. They construct VisPhyBench, a dataset of 209 scenes derived from 108 physical templates, and a pipeline that produces valid reconstructed videos in 97.7% of cases. Experiments using VisPhyBench reveal that while MLLMs demonstrate strong semantic scene understanding, they struggle with accurately inferring physical parameters and simulating consistent physical dynamics.
MLLMs may ace your visual question answering, but VisPhyWorld reveals they're still struggling to actually *simulate* physics.
Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed videos in 97.7% on the benchmark. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics.
