Search papers, labs, and topics across Lattice.
This paper introduces FraQ, an efficient method for coordinate-space recompression in federated Low-Rank Adaptation (LoRA) that addresses the aggregation mismatch problem in federated learning of Large Language Models (LLMs). By factorizing the aggregate into an orthonormal basis and a compact coordinate matrix, FraQ significantly reduces the computational and memory costs associated with dense matrix decomposition while maintaining high accuracy. Experimental results demonstrate that FraQ achieves performance comparable to uncompressed models while minimizing downlink communication and server-side recompression overhead.
Achieving near-uncompressed accuracy with significantly lower communication costs, FraQ revolutionizes federated LoRA efficiency.
Federated fine-tuning with Low-Rank Adaptation (LoRA) enables efficient collaborative adaptation of Large Language Models (LLMs) without centralizing private data. However, LoRA's two-factor parameterization creates an aggregation mismatch across clients: naively averaging the factors does not recover the average of their induced updates. This mismatch can be avoided by forming the exact aggregate in the full weight space and then recompressing it, but decomposing the resulting dense matrix is computationally expensive and memory-intensive. We propose FraQ, an efficient coordinate-space recompression method for federated LoRA. Starting from stacked factors that exactly represent the aggregate, FraQ factorizes it into an orthonormal basis and a compact coordinate matrix. It then recovers the singular spectrum from a small Gram matrix, selects the smallest rank satisfying a prescribed energy threshold, and maps the selected coordinate subspace back through the basis to construct the global adapter. Experiments on text classification and commonsense reasoning benchmarks show that FraQ achieves accuracy close to uncompressed baselines while substantially reducing downlink communication with low server-side recompression overhead.
