Search papers, labs, and topics across Lattice.
This paper introduces KineBench, a novel benchmarking framework for evaluating embodied world models (EWMs) without relying on Inverse Dynamics Models (IDMs), which can introduce errors in action extraction. By utilizing an explicit kinematic grounding pipeline and cascaded visual foundation models to directly extract 6D end-effector poses, KineBench mitigates attribution ambiguity between model inaccuracies and extractor errors. The evaluation across 20 diverse manipulation tasks reveals nonlinear scaling in embodied video generation, offering insights for optimizing future data-scaling strategies.
KineBench reveals that embodied world models exhibit task-complexity-bounded nonlinear scaling, challenging existing assumptions about model performance and data requirements.
Evaluating the physical consistency of embodied world models(EWMs) is a critical open challenge. While closed-loop evaluation via simulator rollouts offers a more faithful assessment of physical plausibility than open-loop alternatives, existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction. Due to the intricate mapping from 2D pixel space to 3D kinematic space, the learned IDMs can be brittle to data outside their training distribution, resulting in unreliable action extraction from the generated videos with novel objects and scenarios. This creates an unavoidable attribution ambiguity between world model inaccuracies and extractor errors. To reduce this ambiguity, we present KineBench, an IDM-free closed-loop benchmark for EWMs, built upon an explicit kinematic grounding pipeline. Given a generated video, KineBench employs cascaded visual foundation models to directly extract 6D end-effector poses from individual frames, which are then executed in a physics simulator for closed-loop validation. Beyond execution-based task success, KineBench incorporates two classical 3D kinematic metrics--Spectral Arc Length (SPARC) and the Maruyama Manipulability Index--to characterize trajectory smoothness and kinematic feasibility from a robot-centric perspective. Built on 20 diverse manipulation tasks in ManiSkill3, KineBench evaluates EWMs across four progressive suites: basic execution, task transfer, visual out-of-distribution generalization, and complexity-conditioned scaling. Evaluation across frontier models reveals task-complexity-bounded nonlinear scaling in embodied video generation, providing empirical guidance for future data-scaling strategies.
