Search papers, labs, and topics across Lattice.
This study investigates the concept of "Permission Literacy" in mobile GUI agents, focusing on their ability to grant permissions based on task relevance and privacy risk. By employing a four-level permission framework and evaluating multimodal large language models through real GUI tasks with Android-style permission popups, the researchers reveal significant biases in permission granting influenced by task context and requester identity. The findings highlight a critical App-Trust Bias and Task-Prior Override, suggesting that separating task execution from permission authorization could improve decision-making in mobile agents.
Task context and requester identity can drastically alter permission grants in mobile GUI agents, with one change reducing approvals from 26 to 0 in a critical task.
Mobile GUI agents routinely encounter system permission dialogs during task execution, yet their ability to grant only permissions that are necessary for the delegated task remains largely unexamined. We present a systematic study of this capability, which we term Permission Literacy. We construct a four-level permission framework based on task relevance and privacy risk and validate the evaluated scenarios with three independent experts in GUI-agent safety. We inject Android-style permission popups into real GUI tasks and evaluate four frontier multimodal large language models using synchronized annotated screenshots and UI-tree hierarchies, making the requester, permission, justification, and available actions accessible to the agent. Beyond the main study, we conduct controlled interventions that separately vary task context and agent-visible requester identity. Under the same Calendar task, changing only the requester from Calendar to PiMusic reduces grants from 26/32 to 0/32, revealing a strong but task-conditioned App-Trust Bias. Holding a popup fixed while changing task context also substantially changes authorization decisions, revealing a systematic Task-Prior Override. Prompt interventions can reduce unnecessary grants, but their effectiveness is inconsistent across models and may come at the cost of suppressing legitimate grants. These results suggest that separating task execution from permission authorization is a promising design direction for future work.
