Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper formalizes a new class of vulnerability in GUI agents called Visual Atomicity Violation, arising from the time gap between observation and action in screenshot-and-click loops, enabling TOCTOU attacks. It characterizes three attack primitives: Notification Overlay Hijack, Window Focus Manipulation, and Web DOM Injection, demonstrating a 100% success rate for action redirection via Window Focus Manipulation. To mitigate these vulnerabilities, the authors propose Pre-execution UI State Verification (PUSV), a layered defense mechanism that re-verifies the UI state before each action, achieving 100% Action Interception Rate against two of the three attack primitives.
Your GUI agent is vulnerable to attack: a 6-second window lets attackers manipulate the UI between observation and action, potentially redirecting clicks with zero visual evidence.
GUI agents that control desktop computers via screenshot-and-click loops introduce a new class of vulnerability: the observation-to-action gap (mean 6.51 s on real OSWorld workloads) creates a Time-Of-Check, Time-Of-Use (TOCTOU) window during which an unprivileged attacker can manipulate the UI state. We formalize this as a Visual Atomicity Violation and characterize three concrete attack primitives: (A) Notification Overlay Hijack, (B) Window Focus Manipulation, and (C) Web DOM Injection. Primitive B, the closest desktop analog to Android Action Rebinding, achieves 100% action-redirection success rate with zero visual evidence at the observation time. We propose Pre-execution UI State Verification (PUSV), a lightweight three-layer defense that re-verifies the UI state immediately before each action dispatch: masked pixel SSIM at the click target (L1), global screenshot diff (L2a), and X Window snapshot diff (L2b). PUSV achieves 100% Action Interception Rate across 180 adversarial trials (135 Primitive A + 45 Primitive B) with zero false positives and<0.1 s overhead. Against Primitive C (zero-visual-footprint DOM injection), PUSV reveals a structural blind spot (~0% AIR), motivating future OS+DOM defense-in-depth architectures. No single PUSV layer alone achieves full coverage; different primitives require different detection signals, validating the layered design.