Search papers, labs, and topics across Lattice.
This paper introduces ForesightSafety-VLA, a diagnostic benchmark specifically designed to evaluate the safety of vision-language-action (VLA) models in embodied intelligence. By defining a comprehensive 13-category safety taxonomy and assessing models across various dimensions of scene structure, language commands, and visual observations, the authors reveal critical insights into the safety limitations of current VLA systems. The findings indicate that even the most advanced policies exhibit significant safety costs and vulnerabilities, particularly in response to variations in structure and visual input, underscoring the importance of integrating safety into the core evaluation of these models.
Even the strongest VLA models face significant safety challenges, with structural and visual variations leading to greater risks than language commands alone.
In embodied intelligence, safety is a prerequisite for reliable robot deployment in the physical world. Current vision-language-action (VLA) models continue to advance toward general-purpose task capability, yet their embodied safety limits remain poorly understood. To address this gap, we introduce ForesightSafety-VLA, a diagnostic benchmark that makes safety the primary evaluation target for VLA systems. We define a 13-category safety taxonomy covering physical interaction safety (Safe-Core), instruction-side safety (Safe-Lang), and perception-side safety (Safe-Vis), and evaluate policies under three controlled dimensions of variation -- scene structure, language command, and visual observation -- so that failure sources can be diagnosed rather than hidden in a single aggregate score. Beyond binary task success, ForesightSafety-VLA measures process-level risk through cumulative safety cost (CC) and risk exposure time (RET), together with a four-quadrant decomposition of safe/unsafe success and failure. We instantiate 66 safety-augmented base scenarios in RoboTwin across 5 embodiments and report results on representative VLA baselines. Across the evaluated baselines, even the strongest policy incurs non-trivial safety cost and unsafe nominal success, while structure and visual variation induce substantially stronger safety degradation than ordinary language variation. These results suggest that embodied safety is tightly coupled to perception, grounding, and control competence rather than being reducible to post-hoc safety filtering alone.
