Search papers, labs, and topics across Lattice.
This study evaluates the performance of computer-use agents executing tasks through graphical user interfaces (GUIs) versus command line interfaces (CLIs) using a matched execution-layer benchmark of 440 desktop tasks. The findings reveal that while the best GUI agent achieves a 59.1% pass rate, skill augmentation for CLI agents significantly boosts their performance to 69.3%, indicating that the observed CLI deficit is primarily due to incomplete skill coverage rather than inherent model limitations. These results highlight distinct execution bottlenecks for each modality, with GUI agents struggling with long-horizon workflows and CLI agents facing challenges in skill scalability.
Skill augmentation can elevate CLI agent performance beyond GUI counterparts, revealing that the true challenge lies in skill coverage rather than model capability.
Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.
