Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
OPSRD addresses the challenge of distilling role-prompted LLM behavior by leveraging privileged teaching context from a fixed expert role during on-policy self-distillation, bypassing the need for reference solutions. The method trains a role-free student model by exposing it to role-conditioned next-token distributions from a frozen teacher on the student's generated prefixes, guiding it towards under-predicted alternatives. Experiments on competition math benchmarks demonstrate significant accuracy improvements over base models without role prompts at inference, with forward KL divergence proving most effective.
Distilling expert LLM behavior is now possible without reference solutions by using a fixed expert's next-token predictions on the student's *prefixes* as privileged context.
Role prompting elicits specialized behavior from large language models through an expert identity, offering a lightweight way to guide reasoning on demanding tasks. However, evaluating or distilling complete role-prompted answers can miss useful next-token preferences when the sampled solution remains incorrect. Transferring these preferences also requires an objective that reaches alternatives the student rarely predicts. We introduce OPSRD, which uses a fixed expert role as privileged teaching context for on-policy self-distillation without reference solutions. A role-free student generates a trajectory, and a frozen instance of the same base model supplies role-conditioned distributions on its exact prefixes, exposing alternatives beyond the sampled continuation. Teacher-weighted forward KL targets alternatives the student underestimates, with clipping to limit individual vocabulary contributions. Supervision is restricted to the highest-entropy half of student positions, concentrating learning where predictions are uncertain. Experiments on three competition-math benchmarks with Qwen3-1.7B, 4B, and 8B show improvements over the base models without role prompts at inference. Forward KL achieves the highest macro-averaged accuracy among the three evaluated divergences at every scale. Code is available at https://github.com/zhansan114514/OPSRD.