Search papers, labs, and topics across Lattice.
This paper introduces two key innovations in knowledge distillation for language models: an offline KD approach that caches the teacher's top-K logits, resulting in a 29% faster training iteration and up to 41% higher throughput, and a fused, chunked KL loss that maintains linear memory usage relative to sequence length, enabling training with context lengths of up to 32,768 tokens. These methods significantly enhance the efficiency of distillation training, making it feasible to conduct large-scale experiments and ablations. The findings suggest that these techniques can drastically improve the deployment of small language models under resource constraints without sacrificing quality.
Offline KD can achieve the same training loss as online methods while being 29% faster, revolutionizing how we approach model distillation efficiency.
Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD). This recovery step largely decides the final quality, yet it is expensive. We present a practitioner's study of how to make distillation training efficient, organised around two systems contributions. First, we show that offline KD (caching the teacher's top-K logits once and training the student against the cache) matches online distillation at near-identical training loss while removing the teacher from memory, running about 29\% faster per iteration, and reaching up to 41\% higher throughput on a single H200 GPU. Second, we introduce a fused, chunked KL loss that never materialises the full vocabulary-sized logit tensor, making peak memory linear in the sequence length. This removes the memory spike that otherwise caps context length and lets us train at four times the context (32{,}768 tokens) on a single GPU. A separate output-head-only toy benchmark isolates the loss kernel and confirms its memory and iteration-rate scaling from 4K to 256K tokens. Together these make large-scale healing and hundreds of ablations affordable. We also report supporting ablations on loss design and sequence packing. We release our chunked-loss implementation: https://github.com/CompactifAI/Full-Chunked-KL-Loss.
