Search papers, labs, and topics across Lattice.
Tsinghua AIAI LaboratoryECNUFudanSJTUUniversity of Science and TechnologyXJTUZJUJun 21, 2026arXiv:2606.22557This paper introduces MacAgentBench, a new benchmark designed to evaluate computer use agents (CUAs) on macOS, addressing the limitations of existing benchmarks that overlook framework capabilities and long-horizon tasks. The benchmark consists of 676 tasks across 25 applications, with a significant portion requiring both GUI and CLI interactions, and employs a deterministic rule-based evaluation with multi-checkpoint scoring. Results indicate that the best-performing configuration, Claude Opus 4.6 on OpenClaw, achieves a Pass@1 score of 73.7%, highlighting the importance of skill libraries over framework design in agent performance.
CUAs can achieve a 73.7% success rate on complex macOS tasks, but the secret to their performance lies in skill libraries, not just framework design.
Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.
