Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces a benchmark-agnostic framework to evaluate the behavioural robustness of Vision-Language-Action (VLA) models in tabletop manipulation tasks by analyzing successful trajectories under perturbations, rather than solely relying on Task Success Rate (TSR). Across three state-of-the-art VLA models and various perturbation conditions, the study reveals that perturbations can significantly alter the execution of successful trajectories, leading to substantial behavioural divergence even when TSR remains comparable. The findings underscore the necessity of incorporating behavioural metrics, such as motion smoothness, efficiency, and gripper actions, for a comprehensive assessment of VLA model robustness.
State-of-the-art VLA models exhibit significant behavioural divergence under perturbation, even when Task Success Rate remains unchanged, revealing a critical blind spot in current robustness evaluations.
Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphasis on evaluating the robustness of VLA models to perturbations. However, this robustness is still predominantly measured through Task Success Rate (TSR). In this work, we propose a benchmark-agnostic evaluation framework to measure the behavioural robustness of models by characterising how successful trajectories are executed under perturbation. We implement this methodology by extending the widely-used LIBERO and LIBERO-Plus benchmarks. Across three state-of-the-art VLA models, four LIBERO task suites and seven perturbation conditions, we evaluate changes in both typical successful behaviour and its variability, including metrics of motion smoothness, efficiency and gripper behaviour. We find that perturbations can alter the behaviour of successful trajectories, a phenomenon which cannot necessarily be inferred from TSR alone. Across LIBERO suites, we identify cases where state-of-the-art VLA models achieve comparable TSR under the same perturbation condition, yet behaviour on successful trajectories diverges substantially. Therefore, to have a more robust assessment of task performance, we argue that suitable measures of robustness should capture not only whether a task is completed, but also how the robot behaves while completing it. When evaluating the robustness of VLA models, TSR may be complemented by behavioural evaluation metrics that characterise the nature and variability of successful task execution by robots.