Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces OS-Helix, an end-to-end trained multimodal agent that unifies GUI interaction and code execution within a single action space, addressing the limitations of prompt-level tool composition for native desktop automation. By developing application-artifact composition for generating hybrid training data and a decoupled instruction/evaluator pipeline, OS-Helix learns to effectively coordinate visual and programmatic actions. The agent achieves significant performance gains over baseline hybrid models on the OSWorld-Verified benchmark, demonstrating the efficacy of its learned GUI/code coordination and a novel step-balanced RL optimization strategy.
Native desktop agents can finally bridge the gap between visual GUI interaction and programmatic code execution by learning a unified action space, achieving state-of-the-art performance on complex real-world tasks.
Real desktop automation is neither purely visual nor purely programmatic. Agents must understand user-facing interfaces while efficiently manipulating files, scripts, and system state. However, native GUI/code coordination cannot be obtained by prompt-level tool composition. In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data. We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space. To construct the required supervision, we introduce application-artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality. We cold-start the policy with SFT on filtered hybrid data and further improve it through online RL in live desktop environments. For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization. On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B achieve relative improvements of 35.0% and 33.7% over their matched Qwen3.5 hybrid baselines, while OS-Helix-27B-RL further improves by 42.9% to reach 66.44% success. Ablations provide evidence consistent with contributions from learned GUI/code coordination, scalable verifiable supervision, and action step balanced optimization, while demonstrating cross-platform transfer on WindowsAgentArena. Our project homepage is at https://os-copilot.github.io/OS-Helix/.