Search papers, labs, and topics across Lattice.
This paper introduces a standardized benchmark for evaluating Vision-Language-Action (VLA) models on the low-cost SO-101 robotic platform, addressing the gap in robustness assessments outside of simulations and expensive setups. The benchmark includes four manipulation tasks with a structured failure taxonomy and recovery-aware metrics, allowing for a nuanced analysis of policy performance. Experimental findings reveal that while stronger pretrained VLA models generally outperform imitation learning baselines, execution instability is the primary failure mode, emphasizing the need for robust recovery strategies in real-world applications.
Execution instability is the leading failure mode in low-cost robotic manipulation, revealing critical gaps in VLA model robustness that traditional success metrics overlook.
Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation, yet existing evaluations are primarily conducted in simulation or on expensive robotic platforms, leaving their robustness on affordable real-world robots largely unexplored. We present a standardized real-world benchmark for evaluating representative VLA and imitation learning policies on the low-cost SO-101 robotic platform. The benchmark comprises four representative manipulation tasks together with unified evaluation protocols, enabling systematic comparison under embodiment uncertainty. Using real-world teleoperated demonstrations, we fine-tune and evaluate $蟺_{0.5}$, SmolVLA, Wall-X, and ACT directly on the physical platform. Beyond conventional task success rates, the benchmark incorporates a structured failure taxonomy, semantic- and execution-level failure decomposition, and recovery-aware evaluation metrics to characterize policy robustness. Experimental results show that stronger pretrained VLA policies generally outperform the imitation learning baseline, although performance remains highly task-dependent under low-cost robotic deployment conditions. Execution instability emerges as the dominant failure source, while recovery capability varies substantially across architectures. These results highlight the importance of failure and recovery analysis beyond binary task success and establish SO-101 as a practical benchmark for evaluating embodied AI systems under realistic low-cost robotic deployment conditions.
