Search papers, labs, and topics across Lattice.
This paper introduces "World Models' Last Exam in Physics," a novel measurement-based benchmark designed to rigorously evaluate the physical consistency of video world models across diverse phenomena including mechanics, optics, and electromagnetism. By employing controlled tasks with predefined physical criteria and an evaluator that combines observability screening with quantitative measurements, the benchmark offers an interpretable method for diagnosing physical inaccuracies without relying on reference videos. Experiments on eight leading models reveal pervasive physical inconsistencies, with the top performer scoring only 57.76, highlighting a significant gap in current video generation capabilities.
Most video world models fail basic physics tests, with even the best models exhibiting significant physical inconsistencies across mechanics, optics, and fluids.
Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and planning in embodied AI systems. Existing evaluations often rely on model-based judgments or reference videos, while direct physical tests largely focus on mechanics. We introduce World Models'Last Exam in Physics, a measurement-based benchmark for evaluating physical consistency in video world models. The benchmark comprises 40 controlled tasks spanning mechanics, optics, fluids, thermal and phase-change phenomena, electromagnetism, and surface tension. Each task pairs an initial image and a generation prompt with predefined physical criteria, enabling interpretable tests of observable physical relationships without requiring reference videos. Its evaluator combines task-observability screening with task-specific quantitative physical measurements. Experiments on eight video generation models across 1,280 videos reveal persistent physical inconsistencies and substantial variation across tasks, with the best model achieving an overall score of 57.76 out of 100. Evaluation on synthetic videos with known physical relationships provides evidence for the validity of the measurement module under controlled conditions. The evaluator also achieves higher agreement with human judgments than a direct vision-language model baseline in both within-task rankings and pairwise comparisons. By combining coverage across physical domains with scores grounded in measurable evidence and explicit measurement limitations, the benchmark provides an interpretable basis for diagnosing physical inconsistencies and tracking progress toward physically consistent video world models.
