Search papers, labs, and topics across Lattice.
This paper introduces Hybrid Policy Distillation (HPD), a novel approach to knowledge distillation for large language models (LLMs) that reformulates the distillation process as a reweighted log-likelihood objective at the token level. By integrating forward and reverse Kullback-Leibler divergence, HPD effectively balances mode coverage and mode-seeking, while leveraging both off-policy data and approximate on-policy sampling for enhanced performance. The method shows significant improvements in optimization stability, computational efficiency, and overall performance across various tasks, including long-generation math reasoning and short-generation dialogue and code tasks.
Hybrid Policy Distillation achieves superior performance by harmonizing the strengths of forward and reverse KL divergence, transforming the landscape of knowledge distillation for LLMs.
Knowledge distillation (KD) is a powerful paradigm for compressing large language models (LLMs), whose effectiveness depends on intertwined choices of divergence direction, optimization strategy, and data regime. We break down the design of existing KD methods and present a unified view that establishes connections between them, reformulating KD as a reweighted log-likelihood objective at the token level. We further propose Hybrid Policy Distillation (HPD), which integrates the complementary advantages of forward and reverse KL to balance mode coverage and mode-seeking, and combines off-policy data with lightweight, approximate on-policy sampling. We validate HPD on long-generation math reasoning as well as short-generation dialogue and code tasks, demonstrating improved optimization stability, computational efficiency, and final performance across diverse model families and scales. The code related to this work is available at https://github.com/zwhong714/Hybrid-Policy-Distillation.
