Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces AgentLens, a mobile GUI agent that adaptively employs Full UI, Partial UI, and GenUI modalities to communicate with users during task execution. The system leverages Virtual Display to enable background execution with selective visual overlays, addressing the trade-off between transparency and multitasking in mobile agent interaction. User studies demonstrate that AgentLens is preferred by a large majority of participants and achieves high usability and adoption intent scores.
Users overwhelmingly prefer mobile GUI agents that adaptively visualize task execution, suggesting a path beyond simple foreground/background modes.
Mobile GUI agents can automate smartphone tasks by interacting directly with app interfaces, but how they should communicate with users during execution remains underexplored. Existing systems rely on two extremes: foreground execution, which maximizes transparency but prevents multitasking, and background execution, which supports multitasking but provides little visual awareness. Through iterative formative studies, we found that users prefer a hybrid model with just-in-time visual interaction, but the most effective visualization modality depends on the task. Motivated by this, we present AgentLens, a mobile GUI agent that adaptively uses three visual modalities during human-agent interaction: Full UI, Partial UI, and GenUI. AgentLens extends a standard mobile agent with adaptive communication actions and uses Virtual Display to enable background execution with selective visual overlays. In a controlled study with 21 participants, AgentLens was preferred by 85.7% of participants and achieved the highest usability (1.94 Overall PSSUQ) and adoption-intent (6.43/7).