Search papers, labs, and topics across Lattice.
This paper introduces the Spatial-Frequency Joint-Aware Heterogeneous Knowledge Distillation (SFKD) framework, which addresses the limitations of existing methods that primarily focus on homogeneous models by effectively transferring knowledge across heterogeneous architectures. By leveraging wavelet transforms to decouple spatial information and employing a dual-stream refinement module, SFKD enhances the retention of both global structural semantics and local details in the knowledge distillation process. Experimental results across various benchmark datasets reveal that SFKD significantly outperforms traditional approaches, highlighting its effectiveness in preserving crucial spatial information during knowledge transfer.
Heterogeneous knowledge distillation can retain critical spatial information, leading to superior performance across diverse model architectures.
Most existing knowledge distillation methods focus on homogeneous models (e.g., CNN-to-CNN), thereby overlooking the flexibility and potential of knowledge transfer across heterogeneous models. Due to intrinsic inductive bias discrepancies between heterogeneous models that cause spatial distribution inconsistencies, prior heterogeneous distillation methods often weaken or discard spatial information in heterogeneous representations. However, the spatial information in representations often encodes transferable global structural semantics as well as architecture-specific local details, and therefore should not be directly ignored. To better leverage the spatial information encoded in heterogeneous representations, we propose a Spatial-Frequency Joint-Aware Heterogeneous Knowledge Distillation framework (SFKD). By leveraging the complementary properties of wavelet transform spatial locality and Fourier representations in characterizing global energy distributions, we first apply multi-level discrete wavelet transform to explicitly decouple spatial information. The resulting wavelet sub-bands are further refined by a dual-stream dual-stage refinement module, and finally combined with a Gaussian-filtered frequency loss to selectively capture informative global information. Extensive experiments on multiple benchmark datasets under both homogeneous and heterogeneous models demonstrate the superiority of our method.
