Search papers, labs, and topics across Lattice.
The paper introduces MIRAGE, a novel pipeline for prompt injection attacks against mobile GUI agents that leverages user-generated content regions within mobile applications. MIRAGE synthesizes context-aware payloads and renders them in the application's native style to create realistic adversarial examples. Experiments across ten applications and eleven attack intents demonstrate that all five evaluated VLM agents are vulnerable, achieving attack success rates of 23%-30%, while maintaining high realism scores.
Mobile GUI agents are surprisingly susceptible to prompt injection via realistic, attacker-controlled text embedded within ordinary user-generated content, even without modifying the agent, application, or OS.
Mobile graphical user interface (GUI) agents driven by vision-language models (VLMs) perceive the screen as rendered pixels and choose actions from what they see, so they cannot reliably separate trusted interface elements from user-generated content. We present MIRAGE (Mobile Injection of Realistic Adversarial GUI Examples), a pipeline that turns benign mobile screenshots into prompt-injection samples by placing attacker-controlled text into ordinary user-generated content regions, without modifying the agent, the application, or the operating system. MIRAGE operates in three stages: a Localizer identifies user-controllable regions on the screenshot, a Generator synthesises context-aware payloads and renders them in the application's native style, and a Curator moderates realism and balances the samples across applications, region types, and attack intents. A key challenge is that an injected screenshot must stay visually indistinguishable from genuine user content while still diverting the agent; we address this by separating the stages that control reach, realism, and distributional balance. On a 1,111-sample benchmark spanning ten applications and eleven attack intents, all five evaluated VLM agents are vulnerable, with attack success rates of 23%-30%, and MIRAGE scores higher on human realism ratings than the strongest prior attack (3.02 versus 2.52 out of 5). We further find that per-sample realism and attack success are uncorrelated, so visual-quality filtering alone cannot reliably defend against this threat.
