Search papers, labs, and topics across Lattice.
This paper introduces ReOrder-OPD, a method that enhances on-policy distillation (OPD) by incorporating reliability-aware prompt ordering based on the teacher's probability of reaching a correct answer from a student prefix. By defining prompt-level teacher continuation reliability \( R \) and utilizing a proxy based on ROUGE-5 scores, the authors demonstrate that sorting prompts by their reliability significantly improves OPD performance across various settings. The results indicate that descending-\( R \) training outperforms traditional ordering methods, leading to consistent gains in mathematics and code generation tasks with the Qwen3 and Gemma4 models.
Sorting prompts by their reliability can dramatically enhance the effectiveness of on-policy distillation, leading to superior performance in complex tasks.
On-policy distillation (OPD) applies token-level teacher supervision to student-generated trajectories, but this supervision is not always reliable. Existing methods use local confidence or teacher-student agreement to weight, filter, or truncate the sampled trajectory. These signals do not directly determine whether the teacher can continue a student prefix to a correct answer, and trajectory-level interventions can conflate one rollout's unreliability with low expected training value of its prompt. We define prompt-level teacher continuation reliability $R$ as the teacher's probability of reaching a correct answer from a student prefix, averaged over prefixes and trajectories induced by the current student. Oracle experiments show that high-$R$ prompts yield larger OPD gains and that descending-$R$ training outperforms random and ascending orders on a fixed prompt pool. Because estimating $R$ requires many teacher continuations, we use the maximum ROUGE-5 F1 between one independent student rollout and verifier-correct same-prompt teacher trajectories. Across ten equal-frequency bins of this actual score, mean $R$ rises monotonically, showing that the proxy separates coarse reliability levels. ReOrder-OPD sorts prompts by the proxy, then draws independent on-policy training trajectories for vanilla OPD. It improves every matched aggregate comparison across Qwen3 and Gemma4 mathematics settings and Qwen3 code settings. Gains in all six FiRe-OPD and ExOPD settings show that prompt ordering complements within-trajectory supervision.
