Search papers, labs, and topics across Lattice.
This paper introduces LIBERO-Safety, a parametric benchmark designed to evaluate the operational safety of Vision-Language-Action (VLA) models by generating safety-critical scenarios with high stochasticity. The authors develop a keypose-driven data generation pipeline that produces a large-scale dataset of 19,664 collision-free demonstrations, facilitating a comprehensive evaluation of various VLA and embodied foundation models. The findings highlight a critical tension between generalization and safety, revealing that while diverse training can enhance safety, it is ultimately limited by issues in trajectory synthesis and semantic alignment.
High-diversity training improves safety in VLA models, but sub-optimal trajectory synthesis still hinders task success.
Despite the impressive manipulation capabilities of Vision-Language-Action (VLA) models, their operational safety under strict constraints remains largely unverified. To address this, we introduce a parametric safety benchmark to procedurally generate safety-critical scenarios with comprehensive stochasticity. To overcome the scalability bottlenecks of human teleoperation, we develop a novel keypose-driven data generation pipeline. Leveraging this infrastructure, we curate a large-scale dataset of 19,664 strictly collision-free demonstrations with extensive domain randomization. We then conduct a systematic cross-paradigm evaluation of eight VLA and two embodied foundation models. Our analysis reveals a critical generalization-safety tension: although high-diversity training fosters safer trajectories, task success remains fundamentally bottlenecked by sub-optimal trajectory synthesis and semantic misalignment. By providing a scalable pipeline, a robust dataset, and profound failure-mode insights, LIBERO-Safety establishes a crucial foundation for developing safe and reliable VLA models.
Microsoft Research
Tsinghua AI