Search papers, labs, and topics across Lattice.
This paper introduces CAPED, a context-aware pre-upload exposure control layer designed to mitigate incidental visual privacy exposure in mobile GUI agents by selectively masking sensitive information in screenshots. By analyzing task requirements and screen context, CAPED effectively reduces privacy leakage while maintaining high task utility, achieving a substantial reduction in success-conditioned weighted seeded leakage from 0.766 to 0.268. The findings emphasize the importance of treating screenshot uploads as a nuanced decision, advocating for selective exposure based on task relevance rather than blanket sharing of screen content.
CAPED slashes privacy leakage in mobile GUI agents by over 65% while preserving essential task functionality, redefining how we handle screenshot uploads.
Screenshot-based mobile GUI agents can operate ordinary smartphone apps through the same visual interface as a human user, but this capability also turns every screen observation into a privacy boundary. During normal task execution, screenshots may expose contacts, messages, photos, files, recommendations, health cues, and other sensitive context that is unrelated to the user's request. We call this problem incidental visual privacy exposure. It is difficult to address with existing defenses: text anonymization misses many visual and inferential cues, while generic privacy masking can remove the evidence and controls that a GUI agent needs to complete the task. This paper presents CAPED, a context-aware pre-upload exposure control layer for mobile GUI agents. CAPED is designed as a phone-side protection layer: before screenshots are released to a remote multimodal agent, it extracts task requirements, uses screen context as a privacy prior, parses visible UI elements, and selectively exposes only content needed for the current task while masking incidental private content. We evaluate CAPED on AndroidWorld for broad task utility and with a controlled 28-task seeded privacy evaluation used as a measurement instrument for trajectory-level incidental leakage. In this seeded evaluation, Full CAPED reduces success-conditioned weighted seeded leakage from 0.766 under raw screenshots to 0.268 while preserving high task utility. A broader AndroidWorld run shows a remaining prototype-level utility cost, but the results support the central claim that screenshot upload should be treated as an explicit device--cloud boundary decision, governed by task-driven selective exposure rather than all-or-nothing screen sharing.
