Search papers, labs, and topics across Lattice.
This paper introduces OPID, a novel framework for on-policy skill distillation in reinforcement learning that directly extracts skill supervision from completed trajectories, addressing the limitations of sparse rewards in traditional outcome-based methods. By representing trajectory hindsight as hierarchical skills, OPID effectively combines episode-level and step-level skills to enhance decision-making during critical interactions. Experimental results show that OPID significantly improves agent performance, sample efficiency, and robustness across various tasks compared to both outcome-only reinforcement learning and existing skill-distillation approaches.
OPID achieves a remarkable boost in agent performance by leveraging hierarchical skills extracted from on-policy trajectories, transforming sparse rewards into dense, actionable insights.
Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose \textbf{OPID} (\textbf{O}n-\textbf{P}olicy Sk\textbf{i}ll \textbf{D}istillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.
Tsinghua AI