Search papers, labs, and topics across Lattice.
This study investigates the impact of context design in self-distillation for language models by comparing three feedback conditions: binary rewards, reference solutions, and step-aligned critiques. The results reveal that step-aligned critiques significantly enhance performance, yielding a 16.11-point improvement over binary rewards and a 5.27-point improvement over reference solutions. The findings suggest that aligning feedback with the model's reasoning process is crucial for effective self-distillation, as it preserves correct behavior while targeting specific failure points.
Step-aligned feedback in self-distillation can boost model performance by over 16 points by precisely targeting reasoning failures without disrupting correct outputs.
Conditioning a language model on additional context, such as feedback on a previous attempt, typically improves its response. Self-distillation trains the model to retain this improvement when the context is not present. The method works by matching the model's output distribution under two settings: a student that sees only the question, and a self-teacher that also sees the context. What the model learns therefore depends on what context the self-teacher receives, yet the design of this context remains largely unexplored. We study context design for self-distillation by training a solver on feedback from a frozen critic. We compare three conditions: (i) a binary reward (GRPO), (ii) the reference solution, and (iii) a step-by-step critique aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, outperforming GRPO by 16.11 points and reference-solution-conditioned self-distillation by 5.27 points (Avg@12). Per-token advantage analysis reveals why: step-aligned feedback targets only the tokens where reasoning fails, leaving correct behavior intact. Conditioning on the reference solution, by contrast, pressures the model to change its behavior at every token (even correct steps) because an alternative derivation inevitably differs in phrasing and approach. This suggests that structural alignment between feedback and the solver's reasoning is a key driver of self-distillation effectiveness.
