Search papers, labs, and topics across Lattice.
This paper introduces REOPD, a reliability-adaptive reward extrapolation framework for on-policy distillation (OPD) that addresses the limitations of existing methods by employing a token-level compatibility weight and a batch-level adaptive budget. By dynamically adjusting the extrapolation coefficient for each token based on reliability, REOPD mitigates issues of reward hacking and instability that arise from using a single global coefficient. The results show that REOPD outperforms G-OPD in single-teacher mathematics and across multiple teachers, while maintaining performance in single-teacher code, highlighting its effectiveness in diverse settings.
REOPD's token-wise adaptability allows for more stable and reliable training, reducing the risk of reward hacking while enhancing performance across varied domains.
On-policy distillation (OPD) trains a student on its own trajectories under dense token-level supervision from a teacher. Reward-extrapolation methods such as ExOPD amplify the teacher-reference log-likelihood ratio to move beyond direct imitation, but apply a single global coefficient $\lambda$ to every token. This can drive the student to fit extreme peaks in the implicit reward, causing reward hacking and unstable training, and the optimal $\lambda$ varies across domains, requiring costly sweeps. We propose REOPD, a reliability-adaptive reward extrapolation framework for OPD. REOPD combines a token-level compatibility weight with a batch-level adaptive budget, yielding a token-wise coefficient $\lambda_{b,t}=1+\gamma_b q_t$ that preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions. It requires no verifier, reward model, value model, or extra rollout beyond standard OPD. REOPD outperforms G-OPD on single-teacher mathematics and on both domains in the multi-teacher setting, while matching G-OPD on single-teacher code, demonstrating effective fine-grained reliability adaptation across domains and teacher configurations.
