Search papers, labs, and topics across Lattice.
This paper introduces SkillHarness, a novel framework designed to enable Computer-Use Agents (CUAs) to safely learn and utilize skills in dynamic, interactive environments. By modeling skill learning as a safety-constrained interaction process and employing a skill boundary that incorporates multi-source supervision signals, SkillHarness effectively identifies and maintains safe skills throughout their lifecycle. Experimental results show a 57.1% reduction in the unsafe rate of learned skills and enhanced execution stability compared to existing methods, addressing critical gaps in the reliability of CUAs under adversarial conditions.
SkillHarness cuts the unsafe skill learning rate by over 57%, transforming how CUAs operate in unpredictable environments.
Computer-Use Agents (CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need for continual skill learning during interaction. Recent approaches address this challenge by learning reusable skills from successful trajectories. However, these skill learning methods largely assume static and safe environments, overlooking risks from adversarial interactions (e.g., prompt injections) and environmental dynamics (e.g., pop-ups). In dynamic settings, such assumptions can lead to risky skill learning and brittle execution, undermining the reliability of CUAs. This raises the question: how can CUAs learn and use skills safely in dynamic environments? To address this problem, we propose SkillHarness, a framework for safe skill harnessing in dynamic environments. SkillHarness moves beyond static skill abstractions by modeling skill learning and utilization as a safety-constrained interaction process. Specifically, we introduce the skill boundary that leverages multi-source supervision signals to identify safe skills from interaction trajectories, and construct self-improving safety constraints throughout the skill lifecycle. In addition, SkillHarness introduces selective skill reuse, where tasks are guided to decompose according to context and completed through the selective activation of skill subsets. Our experiments demonstrate that SkillHarness significantly reduces the unsafe rate of learned skills by 57.1% and consistently improves execution stability under dynamic environmental changes, outperforming existing baselines.
