Search papers, labs, and topics across Lattice.
This paper addresses the limitations of on-policy self-distillation (OPSD) in language model post-training, particularly the issue of privilege illusion where the student model behaves as if it has access to privileged information that it cannot utilize during inference. To overcome this, the authors introduce Dual-Anchored Policy Distillation (DAPD), which employs a two-level anchoring mechanism that aligns reference and rollout behaviors through Dual-Path and Dual-Source Anchoring. Experimental results demonstrate that DAPD significantly reduces privilege illusion and consistently outperforms OPSD, achieving an average improvement of +2.00 points across tasks on the Qwen3-4B model, with even greater gains at larger scales.
DAPD eliminates privilege illusion in self-distillation, leading to consistent performance improvements across model scales.
On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.
