Search papers, labs, and topics across Lattice.
The paper introduces GUITestScape, a benchmark for evaluating MLLM agents in exploratory GUI testing, encompassing both interaction and display defects across 61 Android applications. It also presents GUIJudge, an open-set evaluator that decomposes testing trajectories into diagnosable capabilities, enabling evaluation beyond predefined annotations. Experiments show that GUIJudge outperforms baselines and reveals that defect detection is a key bottleneck for current models, which can be alleviated by integrating GUIJudge's verifiers.
Current MLLM agents struggle to find GUI defects, but a new benchmark and evaluator reveals the critical bottleneck is detection, and surprisingly, simply integrating the evaluator's verifiers significantly boosts performance without retraining.
Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation falls short on two fronts. First, existing benchmarks focus almost exclusively on interaction defects, leaving display defects outside the evaluation frame. Second, evaluation protocols are bound to predefined defect annotations, collapsing the testing process into a single end-state judgment that conflates qualitatively distinct failure modes. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types, and introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation beyond predefined annotations, substantially outperforming all baselines. Benchmarking on GUITestScape further reveals that detection remains the critical bottleneck for existing models across both defect types, and that integrating GUIJudge's verifiers into existing agents significantly boosts their detection performance without retraining.
