Search papers, labs, and topics across Lattice.
This paper introduces AgentCIBench, a novel evaluation framework designed to assess the privacy risks posed by computer-use agents (CUAs) when they operate across multiple personal applications. The study identifies three critical failure modes鈥攙isual co-location, task-ambiguity overshare, and recipient misalignment鈥攄emonstrating that 11 out of 15 evaluated agents exhibit a concerning average leakage rate of 67.9% across scenarios. By highlighting these vulnerabilities, the authors advocate for the integration of contextual disclosure testing as a necessary pre-deployment safety measure for CUAs.
A staggering 73% of evaluated computer-use agents leak sensitive information across contexts, underscoring a critical gap in privacy safeguards.
Computer-use agents (CUAs) now act on a user's behalf across personal applications such as email, calendars, and to-do lists. This cross-application access is useful, but it also creates a privacy risk that has been largely overlooked: when an agent works in one context, it can pull in information from another that is inappropriate in that context. Hence, we introduce AgentCIBench, an evaluation harness that turns this risk into executable, deterministically scored scenarios. We target three common failure modes in CUAs: visual co-location, where the agent pulls in prohibited items that sit next to the task target in the UI; task-ambiguity overshare, where the agent dumps dense personal state in response to an under-specified prompt; and recipient misalignment, where the agent sends content to an addressee for whom it is inappropriate. We evaluate 15 frontier agents and find a surprisingly high failure rate: 11 of 15 leak on more than 50% of scenarios, with an average leakage of 67.9%, and the same failures persist when agents act end-to-end in the environment to complete the task. We release AgentCIBench to encourage the development of safer computer-use agents and position contextual disclosure testing as a pre-deployment safety check.
