Search papers, labs, and topics across Lattice.
This paper introduces an active evaluation framework for generalist robot manipulation policies, addressing the challenge of rigorous real-world performance assessment in the face of vast task factor combinations. By treating policy evaluation as a sequential experimental design problem, the method employs a probabilistic surrogate model to adaptively select configurations that maximize information gain about policy performance. The results demonstrate that this approach can reduce the number of required trials by 20-40% compared to conventional random testing methods, enhancing efficiency in identifying failure-prone regions.
Active evaluation can cut down trial requirements by up to 40%, ensuring more efficient and effective assessment of robot policies in real-world scenarios.
Generalist robot manipulation policies trained on large, diverse datasets have shown remarkable promise across a wide range of tasks. However, rigorously evaluating these policies remains a fundamental challenge. Real-world performance depends on a large combinatorial space of task factors including object poses and camera viewpoints, making full, exhaustive evaluation intractable. Additionally, real hardware evaluation is slow and resource-intensive, so current practice is to use narrow test suites that can miss critical failure modes and misrepresent true deployment readiness. We propose an active evaluation framework that addresses this challenge by treating policy evaluation as a sequential experimental design problem. Our approach fits a probabilistic surrogate model over a structured space of task factors and adaptively selects evaluation configurations to maximize information gain over the policy's performance distribution, allowing for sample-efficient characterization of policy behavior across unseen conditions and a systematic identification of failure-prone regions. We conduct 2331 real-world evaluations across 3 tasks with 3 factor variations and find that our approach typically saves the evaluator at least 20-40% of trials compared to typical random testing.
