Search papers, labs, and topics across Lattice.
This paper explores the conversion of autoregressive language models (ARLMs) into diffusion language models (DLMs) using a novel On-Policy Distillation (OPD) approach. By employing self-OPD, the authors effectively mitigate the knowledge loss typically associated with transitioning to a DLM objective and resolve the train-inference mismatch that standard DLMs face. The resulting On-Policy Diffusion Language Model (OPDLM) achieves remarkable data efficiency, requiring between 15x to 7,000x fewer training tokens while maintaining strong performance across various tasks.
Transforming autoregressive models into diffusion models can be achieved with up to 7,000x fewer training tokens, revolutionizing the efficiency of model training.
We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs). Rather than pretraining from scratch, prior work replaces the causal attention in ARLMs with bidirectional attention and then trains the resulting model using a DLM objective. However, these approaches incur two distribution shifts. First, transitioning from a next-token prediction objective to a DLM objective can discard knowledge acquired by the ARLM during training. Second, standard DLMs suffer from a train-inference mismatch, as the training loss is defined on randomly masked sequences rather than the trajectories encountered at inference produced by confidence-based decoding. To address both challenges, we introduce an On-Policy Diffusion Language Model (OPDLM) in which On-Policy Distillation (OPD) is employed for ARLM-to-DLM transformation. Specifically, OPDLM is trained via self-OPD, where the student, an ARLM with bidirectional attention, generates its own trajectories, and the teacher, the original frozen ARLM, distills its knowledge by providing target logits on these trajectories. By training directly in an on-policy manner, OPDLM eliminates the train-inference mismatch in DLMs, while distillation from the original model enhances knowledge retention from the ARLM. Empirical results demonstrate that OPDLM requires 15x to 7,000x fewer training tokens with strong performance across a wide variety of tasks. OPDLM avoids the prohibitive cost of DLM pretraining and positions DLM transformation as a form of ARLM post-training.
