Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to On-Policy Distillation (OPD) that allows for knowledge transfer between models with different tokenizers, overcoming a significant limitation of existing methods. By implementing a precise token-mapping algorithm, the authors demonstrate that their cross-tokenizer OPD is not only feasible but also significantly more compute-efficient than traditional Supervised Fine-Tuning (SFT) across various benchmarks. The findings reveal that this method expands the potential for effective teacher-student pairings in LLM training, enhancing the adaptability and performance of language models.
Cross-tokenizer On-Policy Distillation achieves superior efficiency and flexibility, enabling knowledge transfer between diverse model families without the constraints of shared tokenizers.
On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models. However, existing OPD distillation methods require teacher and student models to share the same tokenizer, restricting the applicability of OPD within the model series. Current mainstream practice typically employs Supervised Fine-Tuning (SFT) on teacher-generated responses for cross-tokenizer distillation, which fails to capture the rich knowledge embedded in the teacher's probability distribution. In this work, we enable the standard on-policy distillation method to operate across model families, ensuring that high-fidelity token-level signals can propagate across different tokenizers with a precise token-mapping algorithm. Extensive experiments show that cross-tokenizer OPD is significantly more compute-efficient than baselines on various benchmarks. Our results unlock a broader range of teacher-student pairs for OPD, opening up new avenues for adapting and enhancing interactions between LLMs.
