Search papers, labs, and topics across Lattice.
Gym-Anything is introduced as a framework that automates the creation of interactive computer-use environments from arbitrary software by framing environment creation as a multi-agent task involving a coding agent and an auditing agent. This framework is used to construct CUA-World, a large-scale benchmark consisting of over 10,000 long-horizon tasks across 200 software applications, and CUA-World-Long, a more challenging subset. Distilling successful trajectories from CUA-World into a 2B vision-language model and employing an auditing VLM at test time improves performance on CUA-World-Long, demonstrating the framework's utility.
Forget toy problems: Gym-Anything lets you turn *any* software into an agent environment, unlocking a world of 10K+ real-world tasks spanning medicine, engineering, and more.
Computer-use agents hold the promise of assisting in a wide range of digital economic activities. However, current research has largely focused on short-horizon tasks over a limited set of software with limited economic value, such as basic e-commerce and OS-configuration tasks. A key reason is that creating environments for complex software requires significant time and human effort, and therefore does not scale. To address this, we introduce Gym-Anything, a framework for converting any software into an interactive computer-use environment. We frame environment creation itself as a multi-agent task: a coding agent writes setup scripts, downloads real-world data, and configures the software, while producing evidence of correct setup. An independent audit agent then verifies evidence for the environment setup against a quality checklist. Using a taxonomy of economically valuable occupations grounded in U.S. GDP data, we apply this pipeline to 200 software applications with broad occupational coverage. The result is CUA-World, a collection of over 10K long-horizon tasks spanning domains from medical science and astronomy to engineering and enterprise systems, each configured with realistic data along with train and test splits. CUA-World also includes CUA-World-Long, a challenging long-horizon benchmark with tasks often requiring over 500 steps, far exceeding existing benchmarks. Distilling successful trajectories from the training split into a 2B vision-language model outperforms models 2$\times$ its size. We also apply the same auditing principle at test time: a separate VLM reviews completed trajectories and provides feedback on what remains, improving Gemini-3-Flash on CUA-World-Long from 11.5% to 14.0%. We release all code, infrastructure, and benchmark data to facilitate future research in realistic computer-use agents.
CMU ML