Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates on-policy distillation (OPD) for multi-turn interactions in agentic tasks, revealing the inefficiencies of fully online OPD due to the need for fresh rollouts and teacher queries. The authors introduce Replayed-Prefix On-Policy Distillation (ReOPD), which leverages pre-collected teacher trajectories to provide dense supervision without requiring new environment interactions, effectively mitigating the prefix trap caused by distribution shifts. Experimental results demonstrate that ReOPD not only maintains or enhances accuracy compared to traditional OPD but also achieves at least a 4× increase in rollout efficiency, transforming costly interactions into reusable resources for scalable distillation.
ReOPD turns the costly process of agent-environment interaction into a reusable offline resource, achieving faster and more efficient multi-turn distillation.
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$\times$ faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.
Microsoft Research