Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces UniSkill, a framework for learning actor-aligned skill proposals in large language model agents by using a shared policy to propose skillbank edits (Add, Update, or No Edit) from observed trajectories. It addresses the challenge of conflating skill benefits with actor improvement during co-evolution by employing contrastive action feedback, which measures the log-likelihood gap of actor actions when replacing retrieved skills with proposed ones, thus avoiding costly additional rollouts. UniSkill achieves state-of-the-art performance on ALFWorld and WebShop, demonstrating stable joint training and effectiveness even with smaller policy backbones.
A novel contrastive feedback mechanism allows LLM agents to learn skill proposals that are aligned with the current actor's behavior without requiring expensive skill evaluations.
Large language model agents can improve across tasks by retaining reusable skills distilled from prior interactions. Recent work jointly optimizes task execution and skill extraction, enabling the policy and skillbank to co-evolve. However, as the actor continues learning, rewarding skill proposals through their reuse in subsequent training steps may conflate skill benefits with actor improvement, while directly testing each proposed skill requires costly additional actor rollouts. In this paper, we introduce UniSkill, which uses a shared policy to interact with the environment and propose skillbank edits (Add, Update, or No Edit) from the resulting trajectories. Specifically, the actor learns from environment rewards, while contrastive action feedback guides skill proposal learning. This feedback provides an actor-alignment signal by measuring how replacing the retrieved skill with a proposed skill changes the current actor's action log-likelihood gap between previously collected successful and failed trajectories from the same task, thereby avoiding new rollouts for each proposal. Since proposal-level feedback may suppress an otherwise appropriate edit operation when the proposed skill content scores poorly, we further apply skill-edit support regularization to preserve exploration. Empirically, UniSkill achieves strong performance, reaching 98.4% success on ALFWorld and 84.7% on WebShop while maintaining stable joint training. Further ALFWorld experiments show that UniSkill remains effective when the shared policy uses a smaller backbone. Our implementation is available at https://github.com/LimOkii/UniSKill.