Search papers, labs, and topics across Lattice.
This paper introduces DOPD, a novel dual on-policy distillation approach that addresses the issue of privilege illusion by dynamically routing token-level supervision between privileged teacher and student policies based on their advantage gaps. By varying the strength and strategy of supervision for each token, DOPD effectively enhances the transfer of capabilities while mitigating the risks associated with information asymmetry. Experimental results across large language models and vision-language models show that DOPD outperforms traditional on-policy distillation methods, demonstrating improved stability, robustness, and performance on challenging tasks.
DOPD reveals that intelligently routing supervision based on advantage gaps can significantly enhance capability transfer in distillation, outperforming conventional methods.
On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.
NUS