Search papers, labs, and topics across Lattice.
This paper introduces Work2Skill, a framework for inducing reusable agent skills from human GUI experience by synchronizing screen recordings, input events, and accessibility tree data. The WorkDemo-350 dataset, comprising 350 human GUI demonstrations, was created to facilitate research in this area and evaluate downstream performance. Agents equipped with skills induced by Work2Skill demonstrated significant improvements in task success rates and scores on benchmarks like MyPCBench and OSWorld-Verified compared to baseline models.
Human GUI interactions can be precisely captured and translated into reusable agent skills, dramatically boosting performance on unseen software and workflows.
Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging. Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details. We thus introduce Work2Skill, a framework that captures human GUI experience for reusable agent skill induction. WorkRecorder provides the framework's infrastructure for collecting this experience, aligning screen recordings, mouse and keyboard events, and available accessibility-tree information. Using WorkRecorder, we construct WorkDemo-350, the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation. Its 350 demonstrations are screened for completeness and task success. We then propose an agentic skill-induction method that inspects evidence on demand to produce reusable skills capturing workflow rules, step dependencies, and completion checks. When the same four execution models are equipped with skills induced by each method under shared execution settings, Work2Skill achieves the highest scores and success rates. Mean score improvements over the corresponding No Skill baseline are 4.67--10.54 points on MyPCBench Bounded Action and 3.11--7.50 points on a 286-task subset of OSWorld-Verified. Our code and data are available at https://github.com/china-qijizhifeng/Work2Skill.
