Search papers, labs, and topics across Lattice.
This study investigates the conditions under which online imitation learning (IL) enhances post-training performance in large language models (LLMs), revealing that its effectiveness hinges on the realizability of the student policy class relative to the expert policy. Contrary to the prevailing belief that error accumulation drives online IL's superiority, the authors demonstrate that in realizable settings, offline IL can achieve expert-level performance. However, in non-realizable contexts, they identify an information-theoretic bottleneck for offline IL and show that online IL can still excel despite significant distributional mismatches.
Online imitation learning can outperform offline methods, but only when the student can effectively represent the expert鈥攔ealizability is key.
Online imitation learning (IL), particularly on-policy distillation, has emerged as a strong LLM post-training approach, often outperforming offline supervised fine-tuning (SFT). Yet a principled understanding of when and why online interaction helps remains unclear. In this work, we challenge the view that error accumulation is the main source of online IL's advantage, and instead show that the benefits of online interaction depend critically on whether the setting is realizable, i.e., whether the student policy class can represent the expert policy. Under realizability, we empirically find that offline IL already matches expert performance. In contrast, in non-realizable (misspecified) settings, we prove that offline IL encounters an information-theoretic bottleneck even when horizon $H=1$, and propose a structural characterization of misspecification relative to the reward, under which online IL provably achieves high performance despite a large distributional mismatch between the expert and student policies.
BAIR
CMU ML
Tsinghua AI