Search papers, labs, and topics across Lattice.
This paper introduces UniDFKD, a novel framework for Data-Free Knowledge Distillation (DFKD) that utilizes architecture-agnostic semantic priors to synthesize high-quality data for knowledge transfer from teacher to student models. By employing three key strategies鈥擟ategorical Semantic Conditioning, Spatial Semantic Anchoring, and Spatial Semantic Distillation鈥擴niDFKD effectively enhances the semantic quality of synthesized data, overcoming the limitations of architecture-specific methods. Experimental results show that UniDFKD achieves a new state-of-the-art performance, improving accuracy by over 20% compared to existing DFKD approaches across various model architectures, including CNNs and Vision Transformers.
Achieving over 20% performance improvement in data-free knowledge distillation, UniDFKD eliminates reliance on architecture-specific priors, paving the way for more robust model training across diverse architectures.
Data-Free Knowledge Distillation (DFKD) transfers knowledge from a pretrained teacher model to a compact student model by synthesizing semantically informative data, eliminating the need for access to the original training dataset. Existing DFKD methods rely heavily on architecture-specific statistical priors (e.g., Batch Normalization statistics) to guide data synthesis, however, such architecture-dependent priors are often absent in modern architectures such as Vision Transformers (ViTs), resulting in degraded semantic quality of the synthesized data and consequently catastrophic performance degradation. In this paper, we propose \emph{UniDFKD}, a unified data-free knowledge distillation framework that replaces architecture-specific statistics with explicit, architecture-agnostic semantic priors. \emph{UniDFKD} governs the entire synthesis-distillation pipeline along three dimensions: (1) Categorical Semantic Conditioning (CSC) defines \emph{what} to synthesize by persistently modulating the generator with language-derived embeddings to capture semantic diversity; (2) Spatial Semantic Anchoring (SSA) dictates \emph{where} evidence belongs by anchoring the teacher's spatial attributions to a Gaussian prior; and (3) Spatial Semantic Distillation (SSD) controls \emph{how} knowledge is transferred by explicitly aligning teacher-student spatial evidence alongside predictions. Extensive experiments across CNNs and ViTs demonstrate that UniDFKD establishes a new state-of-the-art, outperforming existing methods by an average absolute margin of over 20\% in both homogeneous and heterogeneous settings.
