Search papers, labs, and topics across Lattice.
This paper critiques the reliability of existing benchmarks for computer-use agents (CUAs) by identifying significant flaws in their scoring methodologies, including evaluator biases and task design issues. The authors audited 150 failure-scored trajectories across multiple benchmarks, revealing that over 15% of FAIL verdicts were incorrect due to evaluator false negatives and broken tasks. They propose a diagnostic taxonomy for genuine failures and offer design rules for more robust CUA evaluation, emphasizing the need for improved assessment frameworks as CUAs become more prevalent.
Over 15% of benchmark failures for computer-use agents are misclassified, revealing critical flaws in current evaluation methods.
Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3\% of FAIL verdicts are wrong: 10.7\% are evaluator false negatives and 4.7\% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.
