Search papers, labs, and topics across Lattice.
This paper introduces Dynamic Subspace Boosting (Dysco), a method designed to mitigate data-parameter interference in federated learning when using Low-Rank Adaptation (LoRA). By dynamically allocating client-specific LoRA subspaces based on local activations and employing multi-round subspace boosting, Dysco enhances the stability of adapter aggregation across heterogeneous clients. Experimental results demonstrate that Dysco significantly reduces training loss and improves performance across multiple federated learning algorithms, achieving up to a 9-fold reduction in loss and a 4.3% performance increase on clinical tasks without substantial overhead.
Dysco cuts training loss by up to 9 times and boosts federated learning performance by dynamically aligning client-specific subspaces, tackling a critical source of instability in LoRA aggregation.
Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability. This interference is controlled by the alignment between LoRA update subspaces and client activations, suggesting that federated LoRA aggregation should be viewed not only as parameter averaging but also as subspace allocation. We propose Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner. In each round, clients compute activation-insensitive subspaces from local representations and transmit only the resulting bases; the server then constructs client-specific merged subspaces through a closed-form solution that maximizes compatibility with other clients'insensitive directions. To handle representation drift, Dysco performs multi-round subspace boosting to preserve past update directions while adapting to future representations. We provide a convergence analysis that embeds the data-parameter interference as an aggregation-error term in a standard federated optimization bound, and prove that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error. Experiments on controlled synthetic federated tasks and on MIMIC-IV clinical-note classification with Llama-3.2-1B show that Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times relative to baselines under the orthogonal-subspace partition the theory identifies, improves all five tested FL algorithms by up to 4.3% on MIMIC, outperforms recent federated LoRA methods, and adds only 0.9% wall-clock overhead. Our code is available at https://github.com/illidanlab/Dysco.
