Search papers, labs, and topics across Lattice.
The paper introduces HiLoRA, a hierarchical LoRA framework for personalized federated learning that adapts Vision Transformers (ViTs) by placing adapters at root, cluster, and leaf levels to capture global, subgroup, and client-specific knowledge, respectively. HiLoRA employs cross-tier orthogonality and cascaded optimization to separate update subspaces and align each tier with its residual personalized objective. The method also includes a LoRA-Subspace Adaptive Clustering mechanism to infer latent client groups via subspace similarity analysis, facilitating knowledge sharing.
HiLoRA achieves superior personalization in federated learning by adaptively clustering clients based on LoRA subspace similarity, enabling targeted knowledge sharing and outperforming standard LoRA-based methods.
Vision Transformers (ViTs) have been widely adopted in vision tasks due to their strong transferability. In Federated Learning (FL), where full fine-tuning is communication heavy, Low-Rank Adaptation (LoRA) provides an efficient and communication-friendly way to adapt ViTs. However, existing LoRA-based federated tuning methods overlook latent client structures in real-world settings, limiting shared representation learning and hindering effective adaptation to unseen clients. To address this, we propose HiLoRA, a hierarchical LoRA framework that places adapters at three levels: root, cluster, and leaf, each designed to capture global, subgroup, and client-specific knowledge, respectively. Through cross-tier orthogonality and cascaded optimization, HiLoRA separates update subspaces and aligns each tier with its residual personalized objective. In particular, we develop a LoRA-Subspace Adaptive Clustering mechanism that infers latent client groups via subspace similarity analysis, thereby facilitating knowledge sharing across structurally aligned clients. Theoretically, we establish a tier-wise generalization analysis that supports HiLoRA's design. Experiments on ViT backbones with CIFAR-100 and DomainNet demonstrate consistent improvements in both personalization and generalization.
