Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SIR, a novel black box method for executing indirect prompt injection (IPI) attacks on computer use agents (CUAs) by leveraging a library of reusable principles and an iterative feedback loop to enhance attack efficacy. By targeting CUAs at the operating system level and employing a deterministic oracle for scoring, the authors demonstrate a significant increase in attack success rates, achieving improvements from 4% to 24% on Claude Opus 4.8 and from 0% to 28% on Gemini 3.5 Flash. The findings highlight the adaptability of the discovered strategies, which can transfer across different model architectures without requiring additional feedback.
Adversaries can boost their attack success rates on computer use agents from near-zero to over 28% by using a self-improving feedback loop to craft indirect prompt injections.
Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal, and they are increasingly deployed to automate everyday digital tasks. Because they can be exposed to untrusted content while operating, they are vulnerable to indirect prompt injection (IPI), in which an adversary plants instructions in content the agent will read and redirects it toward actions that violate the user's intent. Existing CUA safety benchmarks evaluate fixed injections written by hand, which may underestimate the risk posed by an adaptive adversary. We present SIR, a black box IPI attack that (i) composes stealthy injections from a small library of reusable principles stated in plain language and (ii) wraps composition in an iterative feedback loop that diagnoses the victim's failed trajectories and distills the bypasses into new, named strategies that are reapplied across tasks. Unlike prior red teaming of web agents, we target CUAs at the operating system level and score attacks with a fully deterministic oracle, using checks on filesystem, service, and permission state rather than an LLM judge. On experiment, we evaluate three frontier CUAs. Composing principles with feedback raises the attack success rate over a baseline written by hand, for example from 4% to 24% on Claude Opus 4.8 and from 0% to 28% on Gemini 3.5 Flash, while the benign task still completes. Principles discovered against one model further transfer to a different architecture with no additional feedback.