Search papers, labs, and topics across Lattice.
This paper introduces few-medoids, a straightforward coreset selection method that identifies samples closest to the centroid of each class for few-shot knowledge distillation (KD). By conducting extensive experiments across four datasets and various teacher-student model pairs, the authors demonstrate that few-medoids consistently outperforms random selection and other existing coreset strategies. The findings suggest that this simple approach can effectively replace more complex methods in future coreset selection research.
Few-medoids outperforms traditional coreset selection methods with a remarkably simple approach, challenging the notion that complexity is necessary for effective model training.
Coreset selection aims to identify a small and highly representative subset of a massive dataset for efficient model training. The problem remains challenging even in the few-shot knowledge distillation (KD) setup, where a full-scale pre-trained teacher informs the student network. Typical sample selection strategies often struggle to surpass the random selection baseline. In this paper, we showcase few-medoids, an embarrassingly simple coreset selection strategy that chooses the samples closest to the centroid (average image) of each class. We present extensive KD experiments on four datasets, covering a wide range of image classification problems, and three teacher-student model pairs, comprising both convolutional and transformer networks. Although the proposed method is embarrassingly simple, our empirical results indicate that few-medoids is able to consistently surpass the random selection baseline, as well as the other coreset selection strategies. We therefore consider that few-medoids can be used as a drop-in replacement for commonly-used baselines (e.g. herding or k-center Greedy), in future research on coreset selection. To reproduce the reported results, we publicly release our code at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.
