Search papers, labs, and topics across Lattice.
To address the pervasive blind spot of single-environment evaluations, the authors built JarvisGUI, a dynamic benchmark that measures GUI agent performance across heterogeneous operating systems including Android, Windows, and Ubuntu. The system models GUI tasks as typed input-output transformations, enabling the procedural composition of complex workflows requiring inter-device data transfers and shared state coordination. Empirical evaluations show that state-of-the-art open-source agents fail sharply at cross-platform contextual reasoning and long-horizon dependency management, revealing that current single-device benchmark success is deeply misleading.
Single-device benchmarks mask a critical failure mode: current GUI agents break down rapidly as soon as tasks require maintaining state and transferring intermediate results across operating systems.
Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device workflows and dynamically evaluate agent performance within a unified framework. By evaluating agents in virtual environments spanning multiple operating systems, JarvisGUI reveals that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world workflows, exposing a critical capability gap invisible to existing benchmarks.
