Search papers, labs, and topics across Lattice.
This paper introduces FATE-VLA, a failure-aware test generation approach for evaluating Vision-Language-Action (VLA) models, addressing the limitations of static benchmarks that often miss critical robustness issues. By reframing the evaluation as an active failure-discovery problem, the method utilizes diversity-driven exploration and surrogate models to target high-risk scene regions, resulting in a significant increase in uncovered failures鈥攗p to 29.7% more than traditional methods. The findings highlight a critical need for adaptive testing strategies that better reveal model weaknesses prior to deployment, as evidenced by a notable drop in success rates for the GR00T-N1.6 model from 64.4% to 34.7%.
Static benchmarks underestimate VLA model failures, but a new adaptive testing approach reveals up to 29.7% more critical weaknesses.
Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks that randomly sample task scenes. In high-dimensional embodied spaces, failures are sparse and clustered, so static benchmarking can underestimate robustness risks. We reframe VLA evaluation as an active failure-discovery problem and propose a failure-aware test-generation approach that combines diversity-driven exploration with surrogate models learned from observed executions. The method steers testing toward high-risk yet diverse scene regions. Across four state-of-the-art VLA models, it uncovers substantially more failures (up to +29.7 % over selected baselines) while revealing more diverse failure modes. This mean that, for instance, in the case of GR00T-N1.6, success rate dropped from 64.4% to 34.7%. More broadly, our findings call for a shift in VLA evaluation: from passive measurement on fixed task suites to adaptive, failure-seeking test generation that exposes the structure of model weaknesses before deployment.
