Search papers, labs, and topics across Lattice.
This paper introduces RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), a novel approach that enhances Group Relative Policy Optimization (GRPO) in reinforcement learning by selectively applying on-policy distillation (OPD) to improve learning signals from negative RL-groups. By targeting only the most informative samples and tokens based on teacher confidence and student entropy, RSTG mitigates the performance degradation seen in traditional GRPO+OPD methods. Experimental results show that RSTG significantly outperforms naive combinations, achieving improvements of +4.02% on math tasks and +3.05% on coding tasks, underscoring its effectiveness in optimizing learning from sparse rewards.
Selective distillation can unlock critical learning signals in reinforcement learning, leading to significant performance gains in complex tasks.
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.
