Search papers, labs, and topics across Lattice.
This study systematically investigates the use of world models for evaluating robotic policies, addressing the limitations of traditional real-world rollouts that are costly and time-consuming. By introducing WMBench, a benchmark based on extensive real-robot teleoperation data, the authors conduct a comprehensive analysis of various video world models and action representation schemes across over 324,000 simulated policy rollouts. Key findings reveal that the reliability of world models for policy evaluation is primarily influenced by long-horizon rollout consistency and that architectural choices significantly impact alignment with real-world robot behavior, leading to the development of GigaWorld-1, a world model optimized for this purpose.
Evaluator quality for robotic policies hinges more on long-horizon consistency than on short-term visual fidelity, reshaping our approach to world model design.
Evaluating embodied robot foundation models remains a critical bottleneck; unlike large language models efficiently assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts limited by hardware and human supervision, which has driven interest in world models as surrogate policy evaluators, yet the key properties that make a world model reliable for policy assessment remain poorly understood. This work presents a systematic study of world models for robotic policy evaluation and introduces WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts covering diverse manipulation tasks to enable controlled comparisons across model families, action encodings, rollout horizons, and evaluation metrics. Using WMBench, we analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, further enriching our analysis with large-scale community submissions from the CVPR 2026 GigaBrain Challenge, curated synthetic trajectories, and a training videos spanning more than 12,000 hours. Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism; pretraining gains stem not only from data scale but from balancing general world knowledge with robot-specific controllability; and architectural choices including action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior. Drawing on these results, we derive a practical design roadmap and realize it in \textit{GigaWorld-1}, a world model specially optimized for policy evaluation, and we fully release our code, models, datasets, and toolkits to advance scalable evaluation research for embodied foundation models.
Tsinghua AI