Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the vulnerability of LLM-based computer use agents (CUAs) to indirect prompt injection by introducing SA-CUA, a dual-LLM framework that establishes a trust boundary using accessibility tree information to deterministically identify and isolate trusted regions from privileged LLM screenshots. By separating trusted and untrusted content and employing a robust endorsement channel for inter-domain information verification, SA-CUA enables autonomous task execution with untrusted data while maintaining high utility and zero attack success rate. This approach significantly improves upon prior methods like CaMeL-CUA by providing a reliable mechanism for safe interaction with dynamic, potentially adversarial computer environments.
A single LLM can now autonomously navigate untrusted computer interfaces and execute sensitive tasks without human oversight, achieving zero attack success rate and higher utility than previous methods.
The safety of LLM-based agents currently rests on alignment training and safety classifiers, neither of which can rule out that a sophisticated indirect prompt injection (IPI) goes through and causes significant harm. Recent work therefore builds agents with information-flow guarantees that rely on structured tool interfaces to interact with the environment. Computer Use Agents (CUAs) act visually on the computer screen, which does not provide a boundary between trusted first-party interfaces and untrusted, potentially attacker-authored content. Actions are equally ambiguous: a click may land on an application toolbar or a button controlled by an attacker. Treating the user as the source of trust and confirming each sensitive decision produces confirmation fatigue, with the risk of missing harmful actions. We introduce SA-CUA, a dual-LLM framework that generates a reliable trust boundary for CUAs by deriving conservative deterministic rules from the accessibility tree for unmasking trusted regions from the privileged LLM's screenshot, keeping trusted and untrusted content separated. Human interaction is only required for proper task specification in the beginning. During the task a robust endorsement channel allows values to pass from the untrusted to the trusted domain by verifying information via trusted channels. This allows the agent to autonomously handle tasks involving untrusted information. SA-CUA maintains much higher utility than CaMeL-CUA while keeping the attack success rate at zero.