Search papers, labs, and topics across Lattice.
This paper investigates on-policy distillation (OPD) for multi-turn interactions between a large language model (LLM) agent and its environment, revealing the inefficiencies of fully online OPD due to the need for fresh rollouts and teacher queries. The authors introduce Replayed-Prefix On-Policy Distillation (ReOPD), which leverages pre-collected teacher trajectories to provide dense supervision without incurring additional environment interaction costs. Their results indicate that ReOPD not only maintains or enhances accuracy compared to traditional OPD but also achieves a training speed that is at least four times faster, effectively transforming costly interactions into reusable resources for scalable distillation.
ReOPD turns the costly process of agent-environment interaction into a reusable offline resource, achieving faster training while preserving accuracy.
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$\times$ faster per training step than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.
Microsoft Research