Search papers, labs, and topics across Lattice.
This study introduces MobileJudgeBench, a comprehensive benchmark designed to evaluate the effectiveness of LLM-based judges in assessing mobile agent task completion across various applications. By analyzing 931 human-annotated trajectories and comparing six judge methods, the research finds that a simple baseline using sampled screenshots often outperforms more complex judge systems, with the choice of LLM backend being a significant factor in performance. Additionally, the study establishes that benchmark quality metrics can reliably predict the utility of judges in real-world applications, highlighting distinct failure profiles associated with different LLM backends.
A simple screenshot-based judge often outperforms complex LLM evaluation methods, challenging assumptions about the necessity of intricate judging pipelines.
Mobile agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile agent trajectories remains largely unexamined. We introduce MobileJudgeBench, a benchmark for systematically evaluating LLM-as-judge methods on mobile agent trajectories. Our benchmark comprises 931 human-annotated trajectories spanning 6 mobile agent benchmarks, 4 agent models, and 68 apps. We evaluate 6 judge methods (five adapted from SPA-Bench, A3 with two modes, AndroidArena, and AgentRewardBench, plus a simple baseline we design) across multiple LLM backends. Our experiments reveal three key findings. First, a simple baseline judge with sampled screenshots is competitive with, and often exceeds, purpose-built methods, indicating that more elaborate judge pipelines do not consistently improve judge quality; among competitive methods, the LLM backbone is the primary driver. Second, benchmark quality metrics reliably predict real-world judge utility: they correlate with both agent ranking fidelity for evaluation and downstream performance when judges serve as reward signals for on-policy reinforcement learning. Third, failure analysis across two LLM backends uncovers qualitatively opposite failure profiles, one conservative and the other permissive, linked to the backbone's precision-recall characteristics.
