Search papers, labs, and topics across Lattice.
This paper introduces Visual Skill Cards (VSCs), a novel memory representation that enhances the decision-making capabilities of computer-using agents by integrating visual cues with procedural workflows. By employing a Trace-to-Visual-Skill-Card approach, SkillLens effectively condenses noisy interaction data into actionable insights, allowing for improved grounded GUI action prediction. The results demonstrate significant performance gains, with an 11.6-point increase in Step Success Rate and a 2.9-point increase in Overall metrics for the frozen GPT-5.4-mini executor, and even greater improvements for the student model using CardDistill.
Visual Skill Cards transform raw interaction data into actionable insights, boosting GUI action prediction accuracy by over 11 points.
Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress. Raw interaction traces preserve such information but are long and noisy to condition on, whereas text-only skills often omit the visual state that makes a procedure applicable. We introduce Visual Skill Cards (VSCs), a state-conditioned memory representation that binds reusable procedures with applicability cues, visual evidence, and verification signals. SkillLens constructs VSCs from heterogeneous interaction experience through Trace-to-Visual-Skill-Card and, at inference time, retrieves relevant cards and selectively expands only the evidence needed by a fixed visual-language model executor for grounded GUI action prediction. The same representation also supports CardDistill, which uses VSC evidence as privileged teacher context to train a student that acts without runtime card retrieval. Across Multimodal-Mind2Web and WebLINX-BrowserGym, SkillLens improves the frozen GPT-5.4-mini executor by +11.6 points in Step SR and +2.9 points in Overall, respectively; CardDistill further improves the corresponding student-only Qwen3-VL-2B metrics by +12.0 and +3.2 points.
