Search papers, labs, and topics across Lattice.
This paper introduces I-SDPO, a novel policy optimization method that leverages instance-level adaptive self-distillation to enhance learning from reward differences in Group Relative Policy Optimization (GRPO). By selectively applying self-distillation based on the success of sampled responses, I-SDPO mitigates the biases introduced by persistent imitation, allowing for more effective learning in scenarios where group-relative rewards are uninformative. The method significantly improves performance on the SciKnowEval benchmark, achieving an average accuracy increase from 56.67% to 70.31% across four scientific domains.
I-SDPO boosts policy optimization accuracy by over 13% by intelligently adapting self-distillation based on instance success rates.
Group Relative Policy Optimization (GRPO) learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect. Privileged self-distillation can fill this gap with dense token supervision, yet applying it throughout training creates a different failure mode: the teacher is a biased, low-variance surrogate for the reward objective, so persistent imitation can oppose reward-improving updates after the policy becomes capable of producing successful trajectories. We introduce I-SDPO (Instance-Level Adaptive Self-Distillation Policy Optimization), which treats teacher reliance as capability-dependent. I-SDPO makes one routing decision per input instance and shares it across that instance's rollout group: all-incorrect groups use a privileged self-distillation objective, whereas any-success groups remain intact for GRPO. This design uses imitation only where group-relative rewards are uninformative. A local analysis characterizes when teacher and reward directions align and shows that a non-vanishing biased distillation weight induces an optimization bias floor. The routing rule automatically reduces the expected distillation rate as success probability rises, withdrawing teacher influence without a hand-designed schedule. On SciKnowEval, I-SDPO obtains the best result in all four scientific domains and improves average mean@16 accuracy from 56.67% with GRPO to 70.31%, with a maximum domain gain of 18.24 points.
