Search papers, labs, and topics across Lattice.
This paper investigates the effects of knowledge distillation (KD) when applied to intermediate layers of Convolutional Neural Networks (CNNs), addressing the potential for overfitting in smaller models. The authors find that while distilling knowledge solely from the final layer is often sufficient for classic datasets, fine-grained and data-scarce scenarios significantly benefit from intermediate supervision, with even a single additional distillation point leading to improved performance. Their approach includes a detailed analysis of supervision strategies informed by attention maps and other metrics, demonstrating that appropriately guided intermediate distillation can enhance model efficiency without compromising accuracy.
Intermediate knowledge distillation can dramatically improve performance in data-scarce environments, turning the conventional wisdom on its head.
The growing size of Convolutional Neural Networks has led to increasingly large and costly models. Knowledge Distillation (KD) addresses this by transferring knowledge from a large network (teacher) to a small one (student), also reducing the training data required. KD is traditionally applied only at the network's final output. However, its behaviour when applied at intermediate network layers has received little attention. This raises the question of whether intermediate block-wise KD, which provides supervision throughout the network, could offer an advantage under specific conditions, such as few instances per class, which is common in fine-grained datasets. This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks. Across eleven datasets, we show that on classic datasets, distilling only the last block is sufficient -- and often best--, whereas fine-grained, data-scarce settings benefit substantially from intermediate supervision, with even a single additional distillation point narrowing the gap considerably. We further study how this supervision should be guided, exploring configurations of varying granularity and informed by an explainability analysis based on attention maps, Centered Kernel Alignment, and Grad-CAM, alongside the impact of teacher and student fine-tuning strategies. This work shows that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
