Search papers, labs, and topics across Lattice.
This paper introduces reference-free metrics for evaluating the physical consistency of videos generated by world models, addressing the limitations of existing methods that rely on human judgment or ground-truth references. By leveraging DROID-SLAM and SEA-RAFT, the authors quantify physical inconsistencies and demonstrate that their relative consistency assessment can enhance task success rates by over 8%, thereby reducing the gap between simulated and real-world performance. Additionally, the absolute assessment offers spatio-temporal localization, allowing for visualization of physical artifacts in generated videos.
Filtering videos for physical consistency can boost task success rates by over 8%, bridging the simulation-to-reality gap in video generation.
We introduce reference-free measures for evaluating the physical consistency of generated videos, combining relative and absolute approaches to assess fidelity. Although tools like WorldGym or WorldEval enable robotic simulation via video generation, physical fidelity gaps often prevent these environments from accurately reproducing real-world task success rates of VLA models. Unlike existing evaluation methods, which require costly human voting (Elo) or unavailable ground-truth references (FVD), our approach utilizes DROID-SLAM and SEA-RAFT to quantify physical inconsistencies, motivated by WorldScore. Videos filtered using our relative consistency assessment show an improvement in task success rates of over 8%, effectively narrowing the simulation-to-reality gap. Furthermore, our absolute assessment enables spatio-temporal localization, providing visualization of when and where physical artifacts occur.
