Search papers, labs, and topics across Lattice.
This paper benchmarks communication backends (MPI, gRPC, PyTorch RPC) in cross-silo federated learning across diverse model sizes and network conditions. It identifies limitations of existing backends for large models in geo-distributed settings and introduces gRPC+S3, a hybrid approach leveraging S3 for model transfer. The results demonstrate that gRPC+S3 achieves up to 3.8x end-to-end speedup over gRPC for large models, providing practical guidance for backend selection in FL.
Naively applying gRPC to cross-silo federated learning with large models can be a bottleneck; gRPC+S3 offers a surprisingly effective solution.
Federated learning (FL) has emerged as a practical means for privacy-preserving distributed machine learning. FL's versatile design makes it suitable for various training settings, from IoT edge devices in cross-device FL to powerful servers in cross-silo FL. A key consequence of this versatility is the high level of diversity found in the networking configuration of FL applications. Coupled with the rising demand for large-scale models such as large language models, well-informed selection and configuration of communication backends become crucial for ensuring optimal performance in FL systems. This work focuses on cross-silo federated learning, presenting in-depth benchmarks of various communication backends, including MPI, gRPC, and PyTorch RPC. In addition, we introduce gRPC+S3, a hybrid backend designed to overcome the limitations of existing approaches, particularly for transmitting large models across geo-distributed deployments, achieving up to $3.8\times$ end-to-end speedup over gRPC. Our benchmarks examine point-to-point and end-to-end performance for a broad range of model sizes running under realistic network conditions. Our findings provide practical insights for selecting and configuring suitable communication backends tailored to the specific federated learning tasks and network configurations.
