Search papers, labs, and topics across Lattice.
This paper investigates the vulnerability of Vision-Language-Action (VLA) robotic systems to physical-world attacks, specifically through a mechanism termed policy-critical action-to-vision attention hijacking. The authors introduce Attention-Guided Semantic Disruption (AGSD), an adversarial patch that effectively diverts attention and disrupts semantic alignment, leading to significant failures in robotic tasks. To counter this threat, they propose Structure-Aware Robust Fine-Tuning (SARF), which significantly reduces the failure rate of VLA robots under AGSD while maintaining performance in clean conditions, demonstrating a viable defense strategy against such attacks.
A novel defense strategy reduces VLA robot failure rates from 100% to an average of 28.6% against sophisticated physical attacks, highlighting a critical vulnerability in robotic systems.
Vision-Language-Action (VLA) policies promise general robotic manipulation, but their robustness against physical-world attacks remains fragile. In particular, we show that physically realizable adversarial patches can reliably induce failures by triggering a mechanism we call policy-critical action-to-vision attention hijacking, where action-conditioned attention is diverted from task-relevant regions to a localized patch. To demonstrate the threat, we propose Attention-Guided Semantic Disruption (AGSD), an Expectation-over-Transformation (EOT) optimized printable patch that jointly (i) concentrates action-to-vision attention on the patch and (ii) disrupts vision-language semantic alignment, yielding strong cross-task and cross-architecture transfer. To mitigate such attacks, we introduce Structure-Aware Robust Fine-Tuning (SARF), a zero-inference-overhead defense that fine-tunes only the visual encoder using feature anchoring, policy-critical attention correction, and language-guided geometric consistency restricted to semantically relevant regions. On LIBERO, SARF reduces OpenVLA's failure rate under AGSD from 100% to 14.2%-56.8% (28.6% average) across suites while preserving clean performance, and on a real PiPER manipulator it improves average success under AGSD from 23.0% to 65.0%. These results highlight mechanism-level robustness as a practical path to securing VLA robots against physical attention hijacking.
