Search papers, labs, and topics across Lattice.
This paper introduces ARKD, an adaptive reinforcement learning-guided framework for knowledge distillation that utilizes both forward and reverse KL divergence to enhance text generation quality. By dynamically adjusting the weights of these divergences based on the characteristics of teacher and student distributions, the method achieves better alignment on both primary and long-tail modes. Experimental results show that ARKD consistently outperforms existing distillation techniques, achieving notable improvements in Rouge-L and BertScore metrics across various benchmarks.
Balancing forward and reverse KL divergence in knowledge distillation can lead to significant improvements in text generation quality, outperforming traditional methods by up to 0.6 points.
Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization. To address this, we analyze the complementary roles of forward and reverse KL divergence (FKL/RKL) in distribution alignment from theoretical and empirical perspectives. We then propose a reinforcement-learning-based adaptive KL-weighted distillation framework, in which a policy network dynamically assigns weights to FKL and RKL based on teacher-student distributional characteristics, guided by immediate reward signals to achieve dual alignment on principal and long-tail modes. Extensive experiments demonstrate consistent improvements across Rouge-L and BertScore metrics, surpassing greedy heuristics by 0.4-0.6 points and outperforming other baseline methods on diverse benchmarks.
