Search papers, labs, and topics across Lattice.
This paper investigates the feasibility of mining explicit skill libraries from interaction data to enhance the performance of computer-using agents. Through a three-stage pipeline that segments GUI trajectories, clusters them into candidate skills, and trains a skill-aware policy, the authors find that while the mined clusters exhibit high purity against existing labels, they do not significantly improve downstream policy performance. The study ultimately reveals limitations in the current methods, suggesting that while trajectory mining can reveal skill structures, it falls short in facilitating reliable cross-domain policy enhancements.
Trajectory mining reveals skill structures but fails to translate these insights into meaningful performance gains for downstream policies.
Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clusters are readable on the source benchmark: five of eight clusters have at least 0.95 purity against InteraSkill Workflows labels. However, readability does not imply transfer. GRPO improves IW skill-step accuracy only from 18.5\% to 20.5\%, leaves BrowseComp+ essentially unchanged, and underperforms trivial frequency priors on key source-domain metrics. We therefore present the method as a diagnostic study: trajectory mining can expose inspectable skill structure, but the current boundary detector, orderless segment representation, and offline reward model are insufficient for reliable cross-domain policy improvement.
MIT CSAIL