Search papers, labs, and topics across Lattice.
This paper introduces Shift-Augmented Knowledge Distillation (SAKD), a novel framework that enhances knowledge distillation by utilizing the evolving features of the student model to generate dynamic perturbations. By shifting the perspective from a static teacher to a more adaptive student-guided approach, SAKD achieves a balance between efficiency and diversity in supervisory signals without the need for multi-stage training. Experimental results on CIFAR-100 and ImageNet show that SAKD outperforms traditional random perturbation methods and matches the accuracy of more complex two-stage approaches while significantly reducing parameter usage and eliminating pre-training requirements.
Shifting the focus from a static teacher to a dynamic student perspective allows for adaptive and diverse view generation in knowledge distillation, achieving superior performance with fewer resources.
Knowledge distillation (KD) typically relies on the fixed perspective of a single teacher, limiting the diversity of supervisory signals. While multi-teacher distillation addresses this by aggregating knowledge from multiple models, it incurs prohibitive computational and storage costs. To balance efficiency and diversity, recent research has focused on generating virtual views from a single teacher. However, existing methods face a trade-off: random perturbation approaches offer efficiency but lack controlled diversity, while structured augmentation methods require multi-stage training and incur linear parameter growth. We observe that this trade-off stems from a common design choice: using the teacher's strong but static features to generate views. Instead, we propose Shift-Augmented Knowledge Distillation (SAKD), a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation. This shift in perspective enables single-stage training while producing adaptive, diverse views through a parameter-free cyclic shift. Extensive experiments on CIFAR-100 and ImageNet demonstrate that SAKD consistently outperforms random perturbation methods and achieves accuracy on par with two-stage approaches, while using significantly fewer parameters and eliminating pre-training requirements.
