Search papers, labs, and topics across Lattice.
The authors introduce WindowsWorld, a new benchmark designed to evaluate GUI agents in complex, multi-application workflows that mirror real-world professional activities. The benchmark comprises 181 tasks across 17 common desktop applications, generated using a multi-agent framework steered by 16 occupations and refined by human review. Experiments with leading large models and agents reveal poor performance on multi-application tasks, particularly those requiring conditional judgment and reasoning across multiple applications, highlighting a significant gap in current agent capabilities.
Today's best GUI agents can barely handle real-world professional workflows, failing at tasks requiring reasoning across just three applications with success rates under 21%.
While GUI agents have shown impressive capabilities in common computer-use tasks such as OSWorld, current benchmarks mainly focus on isolated and single-application tasks. This overlooks a critical real-world requirement of coordinating across multiple applications to accomplish complex profession-specific workflows. To bridge this gap, we present a computer-use benchmark in cross-application workflows, named WindowsWorld, designed to systematically assess GUI Agents on complex multi-step tasks that mirror real-world professional activities. Our methodology uses a multi-agent framework steered by 16 occupations to generate four difficulty-level tasks with intermediate inspection, which are then refined by human review and executed in a simulated environment. The resulting benchmark contains 181 tasks with an average of 5.0 sub-goals across 17 common desktop applications, of which 78% are inherently multi-application. Experimental results of leading large models and agents show that: 1) All computer-use agents perform poorly on multi-application tasks (< 21% success rate), far below the performance of simple single-app tasks; 2) They largely fail at tasks requiring conditional judgment and reasoning across $\geq$ 3 applications, stalling at early sub-goals; 3) Low execution efficiency, where tasks often fail despite far exceeding human step limits. Code, benchmark data, and evaluation resources are available at github.com/HITsz-TMG/WindowsWorld.
