Search papers, labs, and topics across Lattice.
This paper introduces HyperLoRA, a novel framework that enhances federated fine-tuning of foundation models by addressing structural aggregation bias and client-side initialization lag through hypernetwork-driven LoRA generation and product space aggregation. By utilizing a learned generator for client adaptations and a learned aggregation module for synthesizing updates, HyperLoRA significantly improves convergence speed and personalization performance in federated settings. Experimental results demonstrate that HyperLoRA outperforms existing federated LoRA methods, particularly in terms of robustness to distribution shifts and efficiency in heterogeneous client distributions.
HyperLoRA achieves faster convergence and stronger personalization by replacing traditional optimization with learned operators, revolutionizing federated learning efficiency.
Federated fine-tuning of foundation models using Low-Rank Adaptation (LoRA) offers a communication efficient solution for distributed learning. However, existing federated LoRA methods suffer from two fundamental limitations: (1) structural aggregation bias, where independently averaging low rank factors fails to approximate the true combined update, and (2) client side initialization lag, as clients repeatedly reinitialize LoRA parameters across communication rounds, slowing convergence. We propose HyperLoRA, a unified framework that addresses both issues through amortized federated adaptation through hypernetwork-driven LoRA generation and product space aggregation. Instead of iterative per-client optimization, HyperLoRA employs a learned generator that maps client distribution signatures to LoRA initializations, effectively amortizing per client adaptation. On the server side, we introduce a learned aggregation module that directly synthesizes updates in the low-rank product space, eliminating the inconsistencies of factor-wise averaging. A lightweight residual correction module further improves stability under heterogenous (non-IID) client distributions.By replacing iterative optimization and heuristic averaging with learned operators, HyperLoRA jointly enables efficient personalization, unbiased aggregation, and faster convergence. Experiments on federated vision and vision-language benchmarks show that HyperLoRA achieves improved convergence speed, greater robustness to distribution shift, and stronger personalization performance compared to prior federated LoRA methods.
