Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Standard on-policy distillation conditioned on verbal feedback inadvertently transfers arbitrary teacher priors while failing to fully exploit corrective guidance due to flawed target distributions and asymmetric divergence objectives. To resolve this, the authors isolate feedback-derived signals using a probabilistic confirmation framework to construct a student-trust-region target, coupled with an importance-weighted shared-rollout estimator to optimize a symmetric rollout divergence. The resulting approach reliably captures actionable feedback without leaking unmotivated teacher preferences, outperforming both conventional on-policy distillation and recent contrastive baselines on knowledge-based and agentic benchmarks.
Distilling with verbal feedback usually transfers the teacher's arbitrary stylistic biases rather than its genuine corrections—unless you isolate that feedback as formal probabilistic evidence against next-token hypotheses.
Verbal feedback can identify errors and prescribe corrections, providing rich supervision for language-model post-training even when reliable programmatic verifiers are unavailable. Such feedback, often generated by a capable model, can be used to condition the teacher in on-policy distillation, which trains the student to match the teacher's predictions on student-generated rollouts. However, this approach can transfer teacher preferences that the feedback did not motivate, while leaving much of the feedback's guidance unused. We find that both problems come from the standard on-policy distillation objective, specifically the divergence it minimizes and the distribution it uses as its target. Our proposed method addresses both limitations. First, to isolate the information conveyed by the feedback from the teacher's inherent preferences, we treat verbal feedback as evidence for or against the hypothesis that a particular token comes next at a given prefix. We then adopt a probabilistic confirmation framework which uniquely determines an ordering over the vocabulary based on the teacher's predictions before and after it receives feedback. Using a confirmation score consistent with this ordering, we construct a target distribution within a trust region of the student. Second, to learn from guidance that student rollouts can leave unused, we derive a simple shared-rollout estimator of a symmetric divergence between the student and target distributions over rollouts, reusing student and feedback-conditioned teacher rollouts in both directions through importance weighting. Empirical evaluations show that our method outperforms the common on-policy distillation recipe and a recent contrastive variant on knowledge-based and agentic benchmarks.