Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of Knowledge Distillation (KD) in the post-training phase of large language models, particularly focusing on its performance in low-data regimes using the Tulu 3 dataset. The authors find that KD outperforms supervised fine-tuning when data is scarce, but its benefits decrease with more training data; however, distilling from a well-tuned teacher model can significantly enhance performance even with ample data. Additionally, they propose a two-stage KD approach that utilizes synthetic teacher-labeled data followed by human refinement, which consistently boosts student model performance in resource-constrained scenarios.
KD can significantly enhance model performance in low-data settings, but its effectiveness hinges on the quality of the teacher model.
Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model. While prior work has mainly examined task-specific or small-scale settings, the post-training stage for building general instruction-following models has received limited attention. In this paper, we conduct a systematic study of KD in post-training using the large-scale Tulu 3 dataset. We find that KD outperforms supervised fine-tuning (SFT) in low-data regimes, but its advantage diminishes as more training data is added. Distilling from a stronger instruction-tuned teacher restores substantial gains even with abundant data, indicating that KD remains effective when the teacher provides knowledge that the student cannot easily acquire from the training data alone. We further study domain-specific, low-resource scenarios and propose a two-stage KD strategy that leverages synthetic teacher-labeled data followed by refinement on human annotations. This method consistently improves student performance, providing practical guidance for building compact models in data-scarce environments.
