Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces TRISHUL, a spectral-control framework designed to enhance federated parameter-efficient fine-tuning (PEFT) by addressing the challenges posed by non-IID client heterogeneity. By employing shared frozen multi-head low-rank bases and applying nuclear norm proximal shrinkage, TRISHUL effectively suppresses high-variance client-specific updates while ensuring efficient aggregation of model parameters. Experimental results across various benchmarks demonstrate that TRISHUL significantly improves convergence, stability, and overall performance compared to traditional federated LoRA methods, particularly under conditions of increased client heterogeneity.
TRISHUL achieves robust federated fine-tuning by aligning client updates spectrally, leading to superior performance even in highly heterogeneous environments.
Federated parameter-efficient fine-tuning (PEFT) enables communication-efficient adaptation of large pretrained models on decentralized edge data, but it remains fragile under non-IID client heterogeneity. In low-rank adaptation (LoRA), different clients may learn locally useful but spectrally misaligned update subspaces, causing high-variance aggregation and poor global transfer. We propose TRISHUL, a spectral-control framework for robust federated PEFT. TRISHUL follows the FL no-raw-data-sharing setting but does not itself provide formal privacy guarantees. TRISHUL uses shared frozen multi-head low-rank bases to obtain algebraically exact aggregation of compact core updates, applies nuclear norm proximal shrinkage to suppress client-specific high-rank spectral components before upload, and allocates adaptation heads non-uniformly across layers using a concave water filling budget rule derived from pretrained layer capacity. Because shrinkage is performed only on small core matrices, TRISHUL adds negligible computation and no extra per-round communication over the underlying multi-head PEFT protocol. Across vision and language benchmarks, including CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, TRISHUL improves convergence, stability, and final performance over federated LoRA baselines, with greater gains under stronger heterogeneity.