Search papers, labs, and topics across Lattice.
This paper introduces SplitLite, a novel method for communication-efficient split federated fine-tuning of large language models that leverages the low effective rank structure of activation and gradient residuals. By utilizing quantized truncated singular value decomposition (SVD) of these residuals, SplitLite significantly reduces the communication overhead associated with exchanging high-dimensional data between clients and a central server. The results show that SplitLite can decrease activation uplink communication costs by up to 93.5% and total communication costs by up to 83.7%, all while maintaining model performance on the GLUE benchmark.
SplitLite slashes communication costs in federated learning by up to 93.5% without sacrificing performance, revealing a hidden structure in model training data.
Federated fine-tuning of on-device large language models (LLMs) faces a significant computing burden. To overcome this limitation, split learning (SL) has emerged as a promising solution, which offloads the primary training workload to a powerful server. However, SL requires exchanging high-dimensional activations and gradients between clients and the server, resulting in prohibitive communication costs. To overcome this challenge, we propose SplitLite, a communication-efficient split federated LoRA fine-tuning method that exploits the low effective rank structure of consecutive-epoch activation and gradient residuals. Our key finding is that, when LoRA uses rank $r$ updates in parameter space, the activation and gradient residuals of the same data sample between adjacent epochs also exhibit effective rank-$2r$ and rank-$4r$ structures, respectively. By revealing this property, SplitLite transmits only quantized truncated singular value decomposition (SVD) residual factors, thereby significantly reducing both activation uplink and gradient downlink traffic. Extensive experiments on the GLUE benchmark across a series of advanced on-device LLMs demonstrate that our method reduces activation uplink communication costs by up to 93.5\% and total communication costs by up to 83.7\%, without performance degradation.
