Search papers, labs, and topics across Lattice.
This paper introduces Byte-Level Distillation (BLD), a cross-tokenizer distillation method that operates by converting the teacher's output distribution to byte-level probabilities and training a student model with a byte-level decoder head. BLD circumvents the complexities of vocabulary alignment in existing cross-tokenizer distillation methods by using the byte level as a common interface. Experiments across a range of distillation tasks and model sizes (1B-8B parameters) show that BLD performs competitively with, and sometimes surpasses, more complex methods, suggesting the byte level is a viable common ground for cross-tokenizer knowledge transfer.
Bypassing the headache of aligning mismatched vocabularies, byte-level distillation offers a surprisingly simple and effective approach to transferring knowledge between language models using different tokenizers.
Cross-tokenizer distillation (CTD), the transfer of knowledge from a teacher to a student language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher's output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with--and on several benchmarks surpasses--significantly more sophisticated CTD methods, across a range of distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and benchmarks remain elusive, underscoring that CTD is still an open problem.
