Search papers, labs, and topics across Lattice.
This paper introduces ASCEND, an adaptive heterogeneous compression framework for federated knowledge distillation that addresses the challenges of model heterogeneity and communication constraints in federated learning. By formulating the compression strategy selection as a non-stationary stochastic multi-armed bandit problem, the method allows clients to choose from a set of tailored compression strategies based on their unique characteristics and resource capacities. Experimental results show that ASCEND significantly reduces communication overhead and training time while preserving model accuracy across diverse datasets.
Tailoring compression strategies to individual client capabilities can cut communication overhead in federated learning by a significant margin.
Federated learning (FL) enables privacy-preserving distributed model training but faces challenges from heterogeneous model architectures and limited communication resources at the network edge. Federated knowledge distillation (FedKD) alleviates model heterogeneity by combining prototype-wise parameter aggregation and knowledge transfer across heterogeneous models. However, transmitting gradients still introduces considerable communication overhead, while existing compression approaches typically apply a uniform strategy across clients and ignore their diverse model characteristics and resource capacities. To address this issue, we propose a heterogeneous compression framework for FedKD that enables each client to select a compression strategy from a candidate strategy set. We formulate the compression strategy selection problem as a non-stationary stochastic multi-armed bandit (MAB), where each arm corresponds to a compression strategy. An efficiency-aware reward is designed by jointly considering local optimization improvement, global knowledge alignment, and execution time. Based on this formulation, we develop an Adaptive heterogeneouS Compression algorithm for fEderated kNowledge Distillation (ASCEND), which employs an exponential moving average (EMA)-enhanced $蔚$-greedy policy to balance exploration and exploitation. Experimental results on multiple datasets demonstrate that ASCEND effectively adapts to heterogeneous model and resource settings, reducing communication overhead and training time while maintaining competitive model accuracy.
