Search papers, labs, and topics across Lattice.
This paper introduces TurnOPD, a novel approach to on-policy distillation that enhances the training efficiency of long-horizon agents by implementing a turn-level budgeting strategy. By addressing inefficiencies in traditional OPD鈥攕pecifically, the wasteful use of resources on less informative tail turns and the inadequate training of deeper decision points鈥擳urnOPD employs adaptive rollout-depth and progressive turn-normalized loss budgeting. Experiments demonstrate that TurnOPD significantly improves validation accuracy while maintaining equal wall-clock training budgets compared to vanilla OPD, pushing the accuracy-time frontier further.
TurnOPD redefines on-policy distillation by optimizing training budgets at the turn level, leading to superior agent performance without increasing training time.
On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.
Microsoft Research