Search papers, labs, and topics across Lattice.
This paper introduces SeFoRA, a novel algorithm for federated low-rank adaptation (LoRA) that addresses the challenges of heterogeneous client ranks in federated learning. By utilizing linear sketches of local updates, SeFoRA enables effective aggregation of model updates while mitigating bilinear mismatches, thus allowing for efficient fine-tuning in a reduced subspace. Experimental results demonstrate that SeFoRA outperforms existing state-of-the-art methods when fine-tuning RoBERTa-Large on GLUE datasets, achieving convergence at a rate of \(\mathcal{O}(1/T)\) in the rank-homogeneous setting.
SeFoRA enables efficient federated fine-tuning of large models by overcoming the challenges of heterogeneous client ranks, achieving superior performance on benchmark tasks.
We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA,~Hu et al.\ 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate $\cO(1/T)$ for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.
