Search papers, labs, and topics across Lattice.
This paper introduces PRM-as-a-Judge 1.5, a toolkit that enhances robot process assessment by converting rollout videos into detailed progress curves and introducing three new metrics for evaluating robotic performance. These metrics focus on failure-side progress, recovery after setbacks, and execution quality during successful operations, providing a nuanced understanding of embodied model capabilities. The study also presents RoboPulse++, a tool for assessing the reliability of process reward models, alongside a comprehensive user-friendly suite for reproducible evaluations, urging the community to adopt more transparent assessment practices.
Fine-grained metrics reveal that robots can recover from failures more effectively than previously thought, reshaping our understanding of their capabilities.
Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.
DeepMind