Search papers, labs, and topics across Lattice.
The paper formalizes the "visual confused deputy" problem in computer-using agents (CUAs), where agents misperceive screen states and authorize actions on incorrect objects, leading to security vulnerabilities. It demonstrates that simple screen manipulations can redirect routine clicks into privileged actions, highlighting the exploitability of these perception failures. To mitigate this, the authors propose a dual-channel contrastive classification guardrail that independently evaluates the visual click target and the agent's reasoning, blocking execution if either channel indicates risk, and showing improved performance over single-channel approaches.
Even simple screen-level manipulations can trick computer-using agents into performing privileged actions, but a dual-channel guardrail offers a promising defense.
Computer-using agents (CUAs) act directly on graphical user interfaces, yet their perception of the screen is often unreliable. Existing work largely treats these failures as performance limitations, asking whether an action succeeds, rather than whether the agent is acting on the correct object at all. We argue that this is fundamentally a security problem. We formalize the visual confused deputy: a failure mode in which an agent authorizes an action based on a misperceived screen state, due to grounding errors, adversarial screenshot manipulation, or time-of-check-to-time-of-use (TOCTOU) races. This gap is practically exploitable: even simple screen-level manipulations can redirect routine clicks into privileged actions while remaining indistinguishable from ordinary agent mistakes. To mitigate this threat, we propose the first guardrail that operates outside the agent's perceptual loop. Our method, dual-channel contrastive classification, independently evaluates (1) the visual click target and (2) the agent's reasoning about the action against deployment-specific knowledge bases, and blocks execution if either channel indicates risk. The key insight is that these two channels capture complementary failure modes: visual evidence detects target-level mismatches, while textual reasoning reveals dangerous intent behind visually innocuous controls. Across controlled attacks, real GUI screenshots, and agent traces, the combined guardrail consistently outperforms either channel alone. Our results suggest that CUA safety requires not only better action generation, but independent verification of what the agent believes it is clicking and why. Materials are provided\footnote{Model, benchmark, and code: https://github.com/vllm-project/semantic-router}.
