Search papers, labs, and topics across Lattice.
This paper introduces UCOB, a novel framework that enhances agentic reinforcement learning by employing credit-aware on-policy bidirectional self-distillation to effectively utilize and evolve skill memories. By treating skill-conditioned and no-skill prompts as dual on-policy context views, UCOB leverages comparative return-to-go signals to refine skill usage and guide memory updates, resulting in significant performance improvements. Experiments demonstrate that UCOB surpasses existing methods, achieving up to 23.5 and 18.0 point gains over state-of-the-art baselines in complex environments like ALFWorld and WebShop.
UCOB achieves unprecedented performance in agentic reinforcement learning by dynamically refining skill usage through credit-aware self-distillation.
Skill memories can improve agentic reinforcement learning by reusing past experience as textual guidance, but retrieved skills are not oracular: they may help in one state while misleading the same policy in another. This makes the common privileged-teacher assumption fragile, namely that a skill-conditioned prompt can be treated as a fixed teacher for the no-skill prompt. We introduce UCOB, a framework for learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-distillation. UCOB treats skill-conditioned and no-skill prompts as two on-policy context views of the same model, compares their return-to-go within the same task and anchor state, and uses the higher-return view as the local teacher. This local credit signal internalizes useful skill-conditioned behavior, corrects misleading skill usage, and guides task/state skill memory updates, utility-aware retrieval, and reflection self-training. Experiments on agentic tasks, including ALFWorld, WebShop, and Search-QA, show that UCOB outperforms skill-free RL, skill-memory baselines, and self-distillation methods across model scales, with up to 23.5 and 18.0 point gains over SOTA baselines on ALFWorld and WebShop. Ablations and analyses further validate its core mechanisms and efficiency.
