Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
ViSkill addresses the limitations of text-centric skill augmentation in VLM agents by proposing a visual-native framework that encodes successful interactions as reusable visual skill cards. This approach enables direct visual retrieval and utilization of skills for inference and reward shaping, fostering a closed feedback loop where skill accumulation and policy improvement mutually reinforce each other. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves a high success rate of 0.89 (0.91 with cold-start), outperforming baselines and converging faster than PPO.
Visual-native skill cards, directly accessible to VLMs, unlock a self-reinforcing loop of policy improvement and skill accumulation, drastically boosting sample efficiency.
Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO. Our code is available at https://github.com/ZJU-REAL/ViSkill.