Search papers, labs, and topics across Lattice.
This paper introduces FedGSA, a geometry-consistent aggregation framework designed to enhance differential privacy in federated Low-Rank Adaptation (LoRA) for language models. By representing client updates as basis-invariant subspaces on the Grassmann manifold, FedGSA effectively mitigates aggregation mismatch and noise distortion that typically arise from traditional Euclidean methods. Experimental results show that FedGSA outperforms existing techniques, achieving an average accuracy improvement of 2.17% and 2.27% under varying privacy budgets on multiple benchmarks.
Geometry-consistent aggregation can boost federated learning accuracy by over 2% while maintaining differential privacy.
Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models. However, integrating differential privacy (DP) into federated LoRA remains challenging: independently perturbing and aggregating its two low-rank matrices can cause aggregation mismatch and the quadratic noise term. Existing methods mitigate these issues by freezing one low-rank matrix but still rely on Euclidean aggregation, which is basis-dependent and may distort the global update. To address this limitation, we propose FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA. FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold. In each communication round, clients extract low-dimensional subspaces capturing dominant update directions and encode them as projection matrices. The server aggregates these representations to estimate a geometry-consistent global update subspace and reconstructs the global LoRA factors within it, reducing distortion caused by basis misalignment, privacy noise, and heterogeneous client updates. We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions. Experiments on four GLUE tasks and a language generation benchmark demonstrate consistent improvements across privacy budgets and degrees of data heterogeneity. In particular, FedGSA improves average accuracy over the strongest baseline by 2.17% and 2.27% under $\epsilon=6$ and $\epsilon=3$, respectively.
