Search papers, labs, and topics across Lattice.
This paper explores the development of multi-task agentic large language models (LLMs) by employing a two-phase distillation approach, which first trains separate reinforcement learning (RL) experts for individual tasks before consolidating them. The authors identify that off-policy distillation struggles in multi-task settings due to the mode-covering nature of forward KL divergence, leading to performance degradation, while on-policy distillation requires strong initialization. Their proposed method successfully combines the strengths of both approaches, achieving performance on par with single-task RL experts across various tasks, including conversational agents and text-based games.
Off-policy distillation fails in multi-task settings, but a two-phase approach combining it with on-policy refinement can achieve single-task expert performance across multiple tasks.
A key step toward artificial general intelligence is to train models that can perform multiple tasks. In this paper, we study how to build such models by first training separate RL experts for individual tasks and then consolidating them via distillation, as an alternative to directly training a single model on mixed tasks. We show that off-policy distillation degrades in multi-task settings due to the mode-covering nature of forward KL: aggregating data from multiple tasks introduces a large number of behavioral modes that can exceed the student's capacity, forcing it to average across behaviors and leading to degraded performance. In contrast, on-policy distillation is mode-seeking but requires strong initialization. Inspired by these observations, we propose a two-phase approach: off-policy distillation followed by on-policy refinement. Evaluation across conversational agents and text-based games confirms that this two-phase approach matches single-task RL expert performance for each individual task, whereas off-policy or on-policy distillation alone fails to match this performance.
Tsinghua AI