Search papers, labs, and topics across Lattice.
This paper investigates data-free multi-teacher distillation by leveraging noise tailored to the characteristics of teacher models, challenging the conventional reliance on available training data. The authors introduce a novel method called IDeaL, which employs decorrelation losses at both patch and image levels to generate optimized teacher-specific samples for distillation. Experimental results reveal that students trained with IDeaL samples achieve performance comparable to those distilled from real images, even outperforming models distilled from a limited subset of ImageNet images.
Tailored noise in multi-teacher distillation can yield student models that rival those trained on real data, even with just 1K images.
Multi-teacher distillation has emerged as a way to combine complementary teacher models into a single student model that exhibits the strengths of all its teachers. The student is trained to mimic the output of the teachers on a set of images, typically the union of the individual teacher's training sets, assuming this data is available. In this paper, we question that assumption and explore alternative options. We first study how far one can go when distilling from teachers fed with different types of noise. Then, we show that information contained in the teachers can be leveraged to tailor the noise for multi-teacher distillation: we propose a method that, thanks to decorrelation losses at both patch and image levels, generates teacher-specific, improved samples optimized for data-free distillation. Experiments show that our most effective samples, IDeaL, lead to strong students that successfully capture complementary information from the teachers, yielding surprisingly competitive results that substantially narrow the gap with students distilled from real images. Moreover, given a limited budget of 1K images for distillation, students distilled using our IDeaL samples match or surpass the performance of those distilled using a 1K-image subset of ImageNet.
