Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To address the inefficiency of pure-GUI agents and the poor scalability of API-augmented systems, this work introduces a framework that trains computer-use agents to dynamically interleave graphical UI actions with command-line interface execution. By curating the HybridCUA-8K dataset of mixed-mode trajectories and applying supervised fine-tuning followed by CLI-aware reinforcement learning, the model learns optimal switching points between modalities. HybridCUA-9B establishes strong cross-platform performance, reaching 53.6% accuracy on OSWorld (+14.8 percentage points over base) alongside a 4.0 percentage point improvement on WindowsAgentArena.
Computer-use agents can bypass brittle, hand-engineered APIs entirely by learning when to substitute mouse clicks with native shell commands, yielding a 14.8-point leap on OSWorld.
Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often inefficient and error prone, or augment GUI interactions with application specific APIs or tools, which require substantial engineering effort and are difficult to scale across applications. We argue that the next generation of CUAs should combine GUI interactions with the command line interface (CLI), leveraging the generality of the GUI and the efficiency of shell commands. A critical challenge, however, is that current models do not know when or how to use the CLI during task execution. To address this challenge, we develop a data construction pipeline that produces three types of trajectories: GUI only, CLI only, and interleaved GUI and CLI trajectories. This pipeline results in HybridCUA-8K, containing 5K hybrid trajectories and 3K verified RLVR tasks. Building on these data, we propose a training framework with two stages: supervised fine tuning on the constructed trajectories, followed by reinforcement learning with our CLI aware rewards that encourages agents to use the CLI selectively and reliably. Experiments show that HybridCUA-9B achieves 53.6% accuracy on OSWorld, improving over the base model by 14.8 percentage points, and improves performance on WindowsAgentArena by 4.0 percentage points. These results demonstrate the effectiveness and cross platform generalizability of the hybrid GUI and CLI paradigm for computer use agents.
Tsinghua AI