Search papers, labs, and topics across Lattice.
This paper introduces Environment-Native Verified Search (ENVS), a novel search-and-filter pipeline designed to enhance trajectory discovery for GUI agents in live desktop environments. By leveraging verified supervision from environment interactions, ENVS significantly improves the efficiency of policy optimization, achieving a pass rate of 30.3 on a diverse 300-task benchmark while reducing computational costs. Additionally, ENVS demonstrates resilience to desktop interruptions, outperforming traditional online reinforcement learning methods and maintaining superior visual-reasoning capabilities across auxiliary tasks.
ENVS not only reduces training costs but also enhances robustness in GUI tasks, achieving a remarkable pass rate despite real-world interruptions.
As multimodal agents move from interface understanding to real software control, successful trajectory discovery in live desktop environments becomes a key challenge. GUI tasks require long-horizon sequences of precise mouse and keyboard actions, while feedback is sparse, delayed, and costly to obtain through VM rollouts. We propose Environment-Native Verified Search (ENVS), a training-time search-and-filter pipeline that uses the environment to construct verified supervision before policy optimization: it branches over behaviorally distinct GUI actions in live OSWorld VMs, verifies successful leaves, and trains from globally balanced step-level supervision. To evaluate robustness under realistic desktop interruptions, we also introduce OSWorld-Noisy, a dynamic benchmark for recoverable desktop interruptions that preserves the original tasks while testing whether agents can refocus, dismiss, wait, or recover under live perturbations. On the 300-task OSWorld pool, ENVS reaches 30.3 pass@8 on original evaluations and 29.0 on OSWorld-Noisy, outperforming matched ARPO-style online RL while reducing compute from 184-192 to 138-153 GPU-hours; even with only 30% of its search data, ENVS reaches 27.0 pass@8, exceeding ARPO from the base model. Training from noisy environments also better preserves visual-reasoning abilities on auxiliary benchmarks, including OSWorld-G Refusal (16.7 vs. 1.9) and BLINK Functional Correspondence (26.2 vs. 23.1).
