Search papers, labs, and topics across Lattice.
This paper introduces Behavior-Skill, a novel benchmark designed to evaluate vision-language-action (VLA) policies in long-horizon mobile manipulation tasks by focusing on executable constituent skills rather than just overall task success. By analyzing 235,492 skill instances across 50 household tasks, the authors reveal that failures are concentrated in specific skills, particularly those involving contact-rich manipulation, highlighting the non-uniformity of skill performance. The benchmark's trajectory-level and skill-level metrics allow for a more granular understanding of policy capabilities, facilitating targeted improvements in VLA systems.
Failures in long-horizon mobile manipulation tasks are not random; they cluster around specific skills, revealing critical bottlenecks in VLA policy performance.
Reliable execution of long-horizon mobile manipulation tasks remains challenging because overall task success depends on the successful completion of multiple constituent skills. Existing benchmarks, however, still rely primarily on full-task rollouts and aggregate task-level metrics, making intermediate failures difficult to observe and analyze. We present Behavior-Skill, a benchmark that reformulates the learning and evaluation of long-horizon tasks around executable constituent skills. It contains 235,492 skill instances from 10,000 demonstrations across 50 household tasks and 34 semantic skill categories. Each instance pairs a skill instruction with an aligned observation-action segment, and is further associated with a restorable intermediate state and a skill success condition to enable independent evaluation under valid preconditions. We further introduce trajectory-level and skill-level metrics to characterize policy capability beyond aggregate task success. Extensive experiments across representative VLA policies including pi0.5 and GR00T on the complete 50-task benchmark show that failures are highly non-uniform across skills, with contact-rich manipulation skills forming persistent bottlenecks. These results demonstrate that Behavior-Skill complements full-task evaluation by exposing intermediate capability profiles for analyzing and improving long-horizon VLA policies. Behavior-Skill is publicly available at https://github.com/nubot-nudt/Behavior-Skill.
