Search papers, labs, and topics across Lattice.
This paper addresses black-box few-shot knowledge distillation, where a student network learns from a black-box teacher using only a few training examples. To improve student performance, they propose a GAN-based approach that adaptively selects high-confidence synthetic images generated by the GAN, based on the teacher's predictions, and uses them to augment the training set. Experiments across seven image datasets demonstrate state-of-the-art results compared to other few-shot KD methods, indicating the effectiveness of the diversity-promoting training scheme.
Augmenting few-shot knowledge distillation with adaptively selected, teacher-confident GAN-generated images dramatically boosts student accuracy.
Knowledge distillation (KD) is a well-known technique to effectively compress a large network (teacher) to a smaller network (student) with little sacrifice in performance. However, most KD methods require a large training set and internal access to the teacher, which are rarely available due to various restrictions. These challenges have originated a more practical setting known as black-box few-shot KD, where the student is trained with few images and a black-box teacher. Recent approaches typically generate additional synthetic images but lack an active strategy to promote their diversity, a crucial factor for student learning. To address these problems, we propose a novel training scheme for generative adversarial networks, where we adaptively select high-confidence images under the teacher's supervision and introduce them to the adversarial learning on-the-fly. Our approach helps expand and improve the diversity of the distillation set, significantly boosting student accuracy. Through extensive experiments, we achieve state-of-the-art results among other few-shot KD methods on seven image datasets. The code is available at https://github.com/votrinhan88/divbfkd.
