Search papers, labs, and topics across Lattice.
This paper introduces Federated Block Coordinate Gradient Descent (FedBCGD) and its accelerated variant (FedBCGD+) to reduce communication overhead in federated learning with large models like Vision Transformers. FedBCGD divides model parameters into blocks, allowing clients to upload only a specific block in each round, while FedBCGD+ incorporates client drift control and stochastic variance reduction for faster convergence. Theoretical analysis demonstrates a 1/N reduction in communication complexity compared to existing methods, where N is the number of parameter blocks, and experiments validate the algorithms' superior performance.
Achieve up to N-fold communication reduction in federated learning by strategically partitioning and selectively uploading model parameter blocks.
Although Federated Learning has been widely studied in recent years, there are still high overhead expenses in each communication round for large-scale models such as Vision Transformer. To lower the communication complexity, we propose a novel Federated Block Coordinate Gradient Descent (FedBCGD) method for communication efficiency. The proposed method splits model parameters into several blocks including a shared block and enables uploading a specific parameter block by each client, which can significantly reduce communication overhead. Moreover, we also develop an accelerated FedBCGD algorithm (called FedBCGD+) with client drift control and stochastic variance reduction. To the best of our knowledge, this paper is the first work on parameter block communication for training large-scale deep models. We also provide the convergence analysis for the proposed algorithms. Our theoretical results show that the communication complexities of our algorithms are a factor 1 /N lower than those of existing methods, where N is the number of parameter blocks, and they enjoy much faster convergence than their counterparts. Empirical results indicate the superiority of the proposed algorithms compared to state-of-the-art algorithms.
