Search papers, labs, and topics across Lattice.
This paper introduces Group-Reflective Self-Distillation (GRSD), a novel approach that enhances reinforcement learning with verifiable rewards by leveraging the policy's own verified rollouts to provide more aligned and discriminative guidance. By reflecting on successful and failed trajectories within an on-policy group, GRSD enables a self-teacher to refine credit assignment at a granular level, improving the learning process. Experimental results across various agentic environments show that GRSD not only outperforms existing methods but also generalizes better to novel tasks, highlighting its effectiveness in training large language model agents.
GRSD transforms how agents learn from their own experiences, leading to significant improvements in both performance and generalization across tasks.
Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents. However, terminal rewards provide only coarse trajectory-level supervision, leaving successful behaviors, recurring mistakes, and incidental choices entangled in the same outcome signal. Existing agentic self-distillation methods enrich sparse supervision with natural-language skills, but skills retrieved externally or extracted from a single trajectory by stronger models may mismatch current experience, exceed the policy's capability, or remain path-specific. We propose Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts. For each prompt, the policy reflects on each verified trajectory in an on-policy group, and a stop-gradient snapshot contrasts the resulting reflections from successful and failed rollouts to construct group-level privileged guidance. Conditioned on this guidance, a self-teacher refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction. Experiments across multiple agentic environments and model scales demonstrate that GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.
