Search papers, labs, and topics across Lattice.
This paper introduces Diverse Image Priors Knowledge Distillation (DIP-KD), a black-box data-free knowledge distillation framework that addresses the limitations of synthetic data diversity and distillation signals. DIP-KD employs a three-phase pipeline involving image prior synthesis, contrastive learning for sample distinction, and distillation via a novel primer student. Experiments across 12 benchmarks demonstrate state-of-the-art performance, highlighting the importance of data diversity for knowledge acquisition in restricted AI environments.
Black-box knowledge distillation can be significantly improved by synthesizing diverse image priors and using contrastive learning to enhance the distinctions between synthetic samples.
Knowledge distillation (KD) represents a vital mechanism to transfer expertise from complex teacher networks to efficient student models. However, in decentralized or secure AI ecosystems, privacy regulations and proprietary interests often restrict access to the teacher's interface and original datasets. These constraints define a challenging black-box data-free KD scenario where only top-1 predictions and no training data are available. While recent approaches utilize synthetic data, they still face limitations in data diversity and distillation signals. We propose Diverse Image Priors Knowledge Distillation (DIP-KD), a framework that addresses these challenges through a three-phase collaborative pipeline: (1) Synthesis of image priors to capture diverse visual patterns and semantics; (2) Contrast to enhance the collective distinction between synthetic samples via contrastive learning; and (3) Distillation via a novel primer student that enables soft-probability KD. Our evaluation across 12 benchmarks shows that DIP-KD achieves state-of-the-art performance, with ablations confirming data diversity as critical for knowledge acquisition in restricted AI environments.
