Search papers, labs, and topics across Lattice.
The paper introduces Dual-Space Weighting and Time-Warped Alignment (DWA-KD), a novel cross-tokenizer knowledge distillation framework designed to improve token-wise and sequence-level alignment between teacher and student models. DWA-KD employs dual-space entropy-based weighting to emphasize informative tokens during distillation and uses Soft Dynamic Time Warping (Soft-DTW) on both embedding and final hidden-state layers for precise sequence alignment. Experiments across various NLP benchmarks demonstrate that DWA-KD surpasses existing state-of-the-art KD methods, highlighting the benefits of its dual-space weighting and time-warped alignment strategies.
Achieve superior knowledge distillation by warping time and space: DWA-KD aligns teacher and student tokenizers with Soft-DTW and entropy-weighted KL divergence, outperforming SOTA methods.
Knowledge Distillation (KD) has emerged as a crucial technique for compressing Large Language Models (LLMs). Although existing cross-tokenizer KD methods have made notable progress, their effectiveness remains constrained by suboptimal alignment across sequence and vocabulary levels. To address these limitations, we introduce Dual-Space Weighting and Time-Warped Alignment (DWA-KD), a novel cross-tokenizer distillation framework that enhances token-wise distillation through dual-space entropy-based weighting and achieves precise sequence-level alignment by leveraging both lexical and semantic information. At the token level, DWA-KD maps teacher representations into the student space and vice versa, performing dual-space KD via Kullback-Leibler divergence (KL). The process is modulated by dual-space weights that up-weight tokens where the student is uncertain and the teacher is confident, thereby focusing learning on informative tokens rather than treating all positions equally. At the sequence level, DWA-KD applies Soft Dynamic Time Warping (Soft-DTW) to both the embedding and final hidden-state layers, enabling robust alignment of lexical and contextual semantics between teacher and student sequences. Extensive experiments across diverse NLP benchmarks demonstrate that DWA-KD outperforms state-of-the-art KD baselines, while ablation studies confirm the complementary contributions of entropy-based token weighting and embedding and final hidden state layer Soft-DTW alignment.
