Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces RoboValue, a novel sim-and-real benchmark designed to comprehensively evaluate robotic value models across four key dimensions: task-state understanding, temporal progress monitoring, failure/recovery reasoning, and value consistency. By employing diagnostic trajectories and instruction variations across 15 simulated and 20 real-world manipulation tasks, RoboValue reveals significant limitations in current models' fine-grained instruction grounding, history-dependent progress tracking, and recovery assessment, even when exhibiting strong overall performance. The benchmark and its associated dataset provide a critical resource for identifying specific areas for value supervision and advancing the reliability of robotic feedback mechanisms.
Even top-performing robotic value models struggle with nuanced instruction grounding and history-dependent progress tracking, exhibiting inconsistent value judgments across valid execution orders.
Robotic value models assess task execution from visual observations and instructions, providing feedback for policy learning and execution monitoring. However, existing evaluations cover only selected aspects of execution understanding, leaving the reliability of value feedback insufficiently tested. We introduce RoboValue, a fine-grained sim-and-real benchmark for unified evaluation of robotic value models. Its framework evaluates four complementary capability dimensions through shared interfaces: task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency. Diagnostic trajectories and instruction variations test whether value judgments accurately reflect task requirements and changes throughout execution. RoboValue-Dataset covers 15 simulation and 20 real-world manipulation tasks, providing expert demonstrations for training and annotated trajectories for fine-grained evaluation. We evaluate representative robotic value models in zero-shot and one-shot settings under standard conditions and separate shifts in robot embodiment and environment. Our results reveal errors in fine-grained instruction grounding, history-dependent progress tracking, and recovery assessment, even in models with strong outcome and forward-progress scores. The same semantic subtask can also receive substantially different value gains across valid execution orders. These findings identify concrete targets for value supervision, while the dataset and public leaderboard support continued assessment of feedback reliability. The website is available at https://robovalue-benchmark.github.io/.