Search papers, labs, and topics across Lattice.
This paper introduces Bidirectional Context Self-Distillation (BCSD), a novel framework that enhances the ability of reinforcement learning-based skill-based LLM agents to utilize external natural-language skills effectively. By employing a dual-context approach, BCSD evaluates each trajectory from both an augmented view that provides Meta-Skill guidance and a reduced view that emphasizes task-specific skills, leading to improved skill utilization. Experiments on ALFWorld and WebShop show that BCSD significantly outperforms existing methods across various model scales, demonstrating its effectiveness in translating guidance into appropriate actions.
BCSD enables LLM agents to leverage external skills more effectively, achieving superior performance by integrating dual-context evaluations.
External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.
DAMO