Search papers, labs, and topics across Lattice.
This paper introduces CROP, a method that enhances on-policy distillation (OPD) by using counterfactuals to allocate supervision non-uniformly across response tokens based on their task relevance. By constructing original-paraphrase-counterfactual triplets, CROP measures the sensitivity of response positions to task-relevant changes, leading to more effective supervision allocation. The results demonstrate that CROP outperforms traditional selection methods, improving aggregate performance by up to 2.96 points in teacher-student settings, thus validating task relevance as a crucial criterion in selective OPD.
CROP reveals that prioritizing task relevance in token supervision can boost performance by nearly 3 points, challenging conventional selection methods in OPD.
On-policy distillation (OPD) supervises a student language model on trajectories sampled from its current policy, but assigns equal credit to response tokens with unequal supervision value. Selective OPD addresses this limitation by allocating supervision non-uniformly across response tokens according to their estimated training value. Most existing criteria, however, focus primarily on optimization need, such as uncertainty or teacher-student disagreement, while task relevance, namely whether the supervision is tied to the semantic content of the current input, remains less directly characterized as a complementary dimension. To address this gap, we introduce Counterfactual Relevance for On-Policy Distillation (CROP), which operationalizes task relevance through a paraphrase-calibrated counterfactual sensitivity margin. For each source prompt, CROP constructs a validated original-paraphrase-counterfactual triplet, holds the student rollout fixed, and measures each response position by its sensitivity to a task-relevant condition change calibrated by its sensitivity to a meaning-preserving rewrite. Matched selection controls show that CROP identifies more useful supervision positions than random or lowest-relevance selection, while component comparisons confirm the value of both counterfactual sensitivity and paraphrase calibration. Across two teacher-student settings, CROP improves aggregate performance by 1.92 and 2.96 points over the strongest non-CROP selector. These results support task relevance as a complementary criterion for selective OPD and establish CROP as a model-internal, contrast-specific method for allocating token-level supervision.
