Search papers, labs, and topics across Lattice.
The paper introduces HazardArena, a benchmark to evaluate semantic safety in Vision-Language-Action (VLA) models by testing their behavior in safe vs. unsafe "twin" scenarios with identical action requirements. They found that VLA models trained on safe data often fail in unsafe counterparts, revealing a vulnerability where correct action execution leads to unsafe outcomes due to decoupled visual-linguistic semantics. To address this, they propose a Safety Option Layer that uses semantic attributes or a vision-language judge to constrain actions, improving safety with minimal performance impact.
VLA models can ace the task but still trigger unsafe outcomes, exposing a critical gap between action execution and semantic understanding.
Vision-Language-Action (VLA) models inherit rich world knowledge from vision-language backbones and acquire executable skills via action demonstrations. However, existing evaluations largely focus on action execution success, leaving action policies loosely coupled with visual-linguistic semantics. This decoupling exposes a systematic vulnerability whereby correct action execution may induce unsafe outcomes under semantic risk. To expose this vulnerability, we introduce HazardArena, a benchmark designed to evaluate semantic safety in VLAs under controlled yet risk-bearing contexts. HazardArena is constructed from safe/unsafe twin scenarios that share matched objects, layouts, and action requirements, differing only in the semantic context that determines whether an action is unsafe. We find that VLA models trained exclusively on safe scenarios often fail to behave safely when evaluated in their corresponding unsafe counterparts. HazardArena includes over 2,000 assets and 40 risk-sensitive tasks spanning 7 real-world risk categories grounded in established robotic safety standards. To mitigate this vulnerability, we propose a training-free Safety Option Layer that constrains action execution using semantic attributes or a vision-language judge, substantially reducing unsafe behaviors with minimal impact on task performance. We hope that HazardArena highlights the need to rethink how semantic safety is evaluated and enforced in VLAs as they scale toward real-world deployment.
