Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces AmbiBench, a new benchmark designed to evaluate mobile GUI agents' ability to handle ambiguous instructions and engage in interactive intent alignment, moving beyond the limitations of existing benchmarks that focus on one-shot, complete instructions. The benchmark is structured around a taxonomy of instruction clarity levels (Detailed, Standard, Incomplete, Ambiguous) based on Cognitive Gap theory and includes 240 real-world tasks across 25 applications. The authors also present MUSE, an automated evaluation framework using an MLLM-as-a-judge multi-agent architecture, demonstrating its utility in assessing agent performance across different clarity levels and its correlation with human judgment.
Mobile GUI agents struggle with ambiguous instructions, but AmbiBench reveals the performance boundaries of state-of-the-art agents and quantifies the benefits of active interaction for intent alignment.
Benchmarks are paramount for gauging progress in the domain of Mobile GUI Agents. In practical scenarios, users frequently fail to articulate precise directives containing full task details at the onset, and their expressions are typically ambiguous. Consequently, agents are required to converge on the user's true intent via active clarification and interaction during execution. However, existing benchmarks predominantly operate under the idealized assumption that user-issued instructions are complete and unequivocal. This paradigm focuses exclusively on assessing single-turn execution while overlooking the alignment capability of the agent. To address this limitation, we introduce AmbiBench, the first benchmark incorporating a taxonomy of instruction clarity to shift evaluation from unidirectional instruction following to bidirectional intent alignment. Grounded in Cognitive Gap theory, we propose a taxonomy of four clarity levels: Detailed, Standard, Incomplete, and Ambiguous. We construct a rigorous dataset of 240 ecologically valid tasks across 25 applications, subject to strict review protocols. Furthermore, targeting evaluation in dynamic environments, we develop MUSE (Mobile User Satisfaction Evaluator), an automated framework utilizing an MLLM-as-a-judge multi-agent architecture. MUSE performs fine-grained auditing across three dimensions: Outcome Effectiveness, Execution Quality, and Interaction Quality. Empirical results on AmbiBench reveal the performance boundaries of SoTA agents across different clarity levels, quantify the gains derived from active interaction, and validate the strong correlation between MUSE and human judgment. This work redefines evaluation standards, laying the foundation for next-generation agents capable of truly understanding user intent.