Search papers, labs, and topics across Lattice.
This paper addresses the instability issues in Heterogeneous Knowledge Distillation (HKD) by introducing SPOFA, a framework that employs a dual stabilization mechanism for features and gradients. By decoupling feature magnitude from direction and utilizing a momentum-driven dynamic scaler to manage gradient conflicts, SPOFA ensures stable convergence during the distillation process. Experimental results show that SPOFA achieves state-of-the-art accuracy with minimal computational overhead, outperforming existing methods that require more resources.
Achieving state-of-the-art accuracy in knowledge distillation while maintaining a lightweight framework could redefine efficiency benchmarks in model training.
Heterogeneous Knowledge Distillation (HKD) aims to transfer knowledge across varying architectures (e.g., from Transformer to CNN) but inherently suffers from severe training instability. We reveal that this instability stems from two highly coupled challenges: massive feature norm discrepancies that cause optimization drag, and severe gradient conflicts between the primary and distillation objectives arising from distinct inductive biases. To achieve stable distillation, we propose SPOFA, a framework built upon a novel Feature and Gradient Dual Stabilization mechanism. Specifically, at the feature level, we introduce a LayerNorm-based decoupling projector that explicitly decouples feature magnitude from direction, creating a bounded and stable space for semantic alignment. At the gradient level, we propose a momentum-driven Exponential Moving Average (MEMA) dynamic scaler. By establishing a robust historical baseline of the optimization trajectory, MEMA actively evaluates instantaneous gradient conflicts and adaptively penalizes harmful distillation signals, guaranteeing stable convergence. Importantly, SPOFA achieves this dual stabilization with an extremely lightweight parameter footprint. Extensive experiments on two mainstream benchmarks demonstrate that SPOFA achieves state-of-the-art accuracy, significantly outperforming computationally expensive methods while introducing only minimal computational overhead compared to standard baselines.
