Search papers, labs, and topics across Lattice.
This paper introduces SAGE-OPD, a novel framework for multi-turn on-policy distillation that selectively applies teacher supervision based on environment feedback and teacher confidence. By addressing the compounding errors that arise from early mistakes in multi-turn interactions, SAGE-OPD improves the robustness of agent training by dynamically deciding when to intervene or skip teacher guidance. Experiments reveal that SAGE-OPD outperforms standard OPD, achieving up to a 13.3% relative improvement in success rates on unseen tasks, highlighting the importance of selective intervention in multi-turn settings.
Selective teacher intervention in multi-turn training can boost agent performance by over 13% by mitigating the impact of early errors.
On-policy distillation (OPD) improves student models by training them on trajectories induced by their own policy, making it a promising approach for mitigating exposure bias in agent training. However, most OPD studies focus on single-turn settings, while realistic LLM agents interact with environments over multiple turns. In this regime, early errors can alter future observations and compound across the trajectory, and standard dense token-level OPD becomes brittle, as it may over-penalize semantically valid alternatives, reinforce local degeneracies such as repeated actions, and propagate unreliable teacher supervision on off-distribution histories. We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD. Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on. To further address compounding errors, SAGE-OPD weights token-level distillation by teacher confidence, reducing the influence of uncertain teacher distributions on corrupted or ambiguous histories. Finally, SAGE-OPD applies loss normalization to preserve the overall loss scale of standard OPD while retaining selective turn-level weighting. Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD. Ablation studies further demonstrate that turn-level intervention, teacher confidence weighting, and loss normalization provide complementary benefits. Our results suggest that effective multi-turn OPD should remain on-policy, but teacher supervision should be selectively allocated to turns where intervention is necessary and reliable.
DeepMind
Meta AI