Search papers, labs, and topics across Lattice.
This paper introduces the Physics Question Scene Graph (PQSG), a novel evaluation framework designed to assess the physical plausibility of videos generated by text-to-video models. By leveraging a graph-based hierarchy of questions generated by a vision-language model, PQSG enables fine-grained localization of physical law violations and provides a structured evaluation of generated content against prompts. Validation through the FinePhyEval dataset demonstrates that PQSG correlates more strongly with human assessments than previous methods, highlighting its effectiveness in evaluating physical realism in video generation.
PQSG reveals that state-of-the-art video generation models often misrepresent physical laws, with significant implications for the realism of generated content.
Video generation models are increasingly capable of producing realistic videos, but they still struggle to generate videos that follow basic physical laws. Compounding this is a lack of reliable granular evaluation methods for localizing and specifying physical law violations in videos. We address this by introducing Physics Question Scene Graph (PQSG), a hierarchical question-based evaluation pipeline. PQSG evaluates generated videos by checking their faithfulness to a prompt across objects, actions, and adherence to physical laws using a graph-based hierarchy of questions generated by a vision-language model (VLM), guided by high-quality in-context examples. By representing questions as a graph, PQSG introduces logical dependencies within questions, ensuring that each query is contextually valid. Moreover, PQSG provides granular assessments of which qualities of the video violate physical plausibility constraints. We validate PQSG by creating FinePhyEval, a dataset with physics-based prompts and corresponding generated videos from diverse state-of-the-art video generation models (Sora 2, Veo 3, and Wan 2.1), with each video annotated across multiple categories by humans. Using FinePhyEval, we measure the correlation between PQSG's fine-grained scores and human judgments, showing higher overall correlations than prior work. We also find that PQSG ranks closed-source models higher than Wan 2.1 on physical realism. Lastly, we show that the annotations we provide in FinePhyEval can also be used for subtask evaluation: we benchmark two strong VLMs on generating and answering questions, finding that while models can create human-like questions, they still fall short of human performance in answering them.
AI2