Search papers, labs, and topics across Lattice.
This paper introduces Agentic Reinforcement Learning with Self-Distilled Reward Shaping (ADRS), a novel framework that enhances the credit assignment process in multi-turn language agents by leveraging privileged skills for denser supervision. By normalizing token scores and modulating them with a Teacher Value Advantage (TVA) gate, ADRS effectively integrates teacher preferences into the reinforcement learning credit path, addressing the limitations of existing methods that fail to calibrate teacher scores across interaction steps. Experimental results demonstrate that ADRS significantly improves performance on long-horizon tasks across various benchmarks, showcasing its robustness in different settings and with reduced data.
Token-level credit assignment in multi-turn language agents can be effectively enhanced by integrating teacher preferences through a novel self-distillation approach.
Agentic reinforcement learning enables LLM agents to learn through interaction, but sparse trajectory-level rewards reveal success without identifying which intermediate decisions deserve credit. Training-only privileged skills can provide denser supervision by allowing the same frozen policy snapshot to rescore fixed tokens from skill-free trajectories while conditioned on task-matched procedural skills. Existing methods, however, do not jointly calibrate teacher scores across interaction steps, relate teacher confidence to realized returns, and integrate the resulting signal into native reward-to-advantage construction. We introduce Agentic Reinforcement Learning with Self-Distilled Reward Shaping (ADRS), a framework for constructing return-associated token-level credit for multi-turn language agents. ADRS centers and normalizes privileged token scores within each step, modulates them with a return-associated Teacher Value Advantage (TVA) gate based on within-group confidence--return association, and incorporates the gated token signal into native RL credit construction. Together, these components determine what the teacher prefers, when that preference is return-relevant, and how it enters the native reinforcement-learning credit path, while keeping rollouts and inference skill-free. Finally, experiments across three interactive benchmarks show that ADRS consistently improves performance on long-horizon tasks, with gains persisting across RL backbones, reduced-data settings, unseen tasks, and extended training. For anonymous review, our code is available at the following the link: https://github.com/gitrxh/ADRS-arxiv
DAMO