Search papers, labs, and topics across Lattice.
VenusBench-Mobile is introduced as a new online benchmark for mobile GUI agents, designed to reflect the diversity and instability of real-world mobile usage through user-intent-driven tasks. Evaluation of state-of-the-art agents on VenusBench-Mobile reveals significant performance gaps compared to existing benchmarks, highlighting challenges in perception, memory, and robustness to environment variations. Diagnostic analysis pinpoints perception and memory as key bottlenecks, suggesting current agents are far from reliable real-world deployment.
Current mobile GUI agents, despite progress on existing benchmarks, crumble under the realistic demands of user-centric tasks, revealing critical weaknesses in perception, memory, and robustness.
Existing online benchmarks for mobile GUI agents remain largely app-centric and task-homogeneous, failing to reflect the diversity and instability of real-world mobile usage. To this end, we introduce VenusBench-Mobile, a challenging online benchmark for evaluating general-purpose mobile GUI agents under realistic, user-centric conditions. VenusBench-Mobile builds two core evaluation pillars: defining what to evaluate via user-intent-driven task design that reflects real mobile usage, and how to evaluate through a capability-oriented annotation scheme for fine-grained agent behavior analysis. Extensive evaluation of state-of-the-art mobile GUI agents reveals large performance gaps relative to prior benchmarks, indicating that VenusBench-Mobile poses substantially more challenging and realistic tasks and that current agents remain far from reliable real-world deployment. Diagnostic analysis further shows that failures are dominated by deficiencies in perception and memory, which are largely obscured by coarse-grained evaluations. Moreover, even the strongest agents exhibit near-zero success under environment variations, highlighting their brittleness in realistic settings. Based on these insights, we believe VenusBench-Mobile provides an important stepping stone toward robust real-world deployment of mobile GUI agents. Code and data are available at https://github.com/inclusionAI/UI-Venus/tree/VenusBench-Mobile.
