Search papers, labs, and topics across Lattice.
This paper critiques the standard success rate metric used to evaluate Vision-Language-Action (VLA) models on long-horizon robotics benchmarks like BEHAVIOR1K, arguing it overlooks critical safety and robustness aspects. They conduct a detailed analysis of state-of-the-art VLA models, evaluating them on reproducibility, consistency, safety, and task awareness. The authors find that current metrics exaggerate performance and propose new evaluation protocols to better capture safety violations in complex, interactive scenarios.
Current VLA benchmarks may be overstating real-world readiness, as models succeeding by standard metrics often exhibit unsafe behaviors and poor robustness.
Vision-language-action models (VLAs) have been extensively used in robotics applications, achieving great success in various manipulation problems. More recently, VLAs have been used in long-horizon tasks and evaluated on benchmarks, such as BEHAVIOR1K (B1K), for solving complex household chores. The common metric for measuring progress in such benchmarks is success rate or partial score based on satisfaction of progress-agnostic criteria, meaning only the final states of the objects are considered, regardless of the events that lead to such states. In this paper, we argue that using such evaluation protocols say little about safety aspects of operation and can potentially exaggerate reported performance, undermining core challenges for future real-world deployment. To this end, we conduct a thorough analysis of state-of-the-art models on the B1K Challenge and evaluate policies in terms of robustness via reproducibility and consistency of performance, safety aspects of policies operations, task awareness, and key elements leading to the incompletion of tasks. We then propose evaluation protocols to capture safety violations to better measure the true performance of the policies in more complex and interactive scenarios. At the end, we discuss the limitations of the existing VLAs and motivate future research.
