Search papers, labs, and topics across Lattice.
This paper investigates a novel vulnerability in vision-language-action (VLA) models, where seemingly benign prompts can redirect the final outcomes of robotic actions without altering the perceived task. The authors formalize this threat as command-preserving trajectory redirection, enabling attackers to manipulate robot behavior by subtly modifying prompts while maintaining their original appearance. Through an innovative on-policy prompt search method, the study demonstrates that these near-benign perturbations can effectively guide VLA systems to unintended targets, revealing significant security implications for robot control systems.
Seemingly innocuous prompts can covertly hijack robotic actions, steering them toward adversarial outcomes while maintaining the facade of intended commands.
Vision-language-action (VLA) policies bring natural language into closed-loop robot control, enabling robots to execute manipulation tasks directly from text instructions. The same interface gives text a recurring role in control because the prompt is reused at every replanning step, and each prompt-conditioned action changes the future observations on which the policy acts. Existing VLA attacks study adversarial prompts that elicit targeted low-level actions or make such actions persist across changing images. We identify a stronger trajectory-level failure mode: a prompt that still $\textit{appears}$ to specify the intended task but redirects the final physical outcome. We mathematically formalize this setting as $\textit{command-preserving trajectory redirection}$, a prompt-only threat model in which the attacker chooses one prompt before the episode, all policy and environment components remain fixed, and the prompt must stay close to the benign instruction while omitting target words and correction language. To find such prompts, we introduce an on-policy prompt search method that uses rollouts to discover perturbations whose closed-loop behavior tracks a target task while satisfying the command-preserving constraints. Experiments in simulation and on hardware show that near-benign prompt perturbations can redirect VLA rollouts to attacker-specified targets. These results expose a trajectory-level vulnerability in VLA instruction grounding: text that appears to preserve the intended command can still give an adversary control over the robot's final physical outcome. Project website: https://vla-redirection-attack.github.io/
