Search papers, labs, and topics across Lattice.
This paper introduces PreLort, a novel nested low-rank formulation for federated fine-tuning of large language models that addresses the challenges posed by heterogeneous hardware resources. By organizing adapter dimensions into a prefix hierarchy and implementing a segment-wise aggregation rule, PreLort optimally utilizes task-relevant information while allowing lower-rank clients to benefit from the richer data of higher-rank clients. Experimental results show that PreLort outperforms existing methods in accuracy and ROUGE-L metrics, while maintaining competitive perplexity across various base models.
PreLort ensures that low-rank clients can leverage the richer information from higher-rank clients, leading to significant improvements in model performance.
Federated fine-tuning of large language models using parameter-efficient methods such as LoRA enables privacy-preserving adaptation of foundation models. Heterogeneous hardware resources introduce challenges, as clients with different adapter ranks cannot be directly aggregated. While existing methods enable aggregation under heterogeneous ranks, they fail to control how information is distributed across rank dimensions, leading to suboptimal use of shared low-rank representations. Instead, we propose PreLort: a nested low-rank formulation for federated LoRA that organizes adapter dimensions into a prefix hierarchy. Our approach ensures that lower-rank dimensions encode task-relevant information, while higher-rank dimensions capture additional capacity. Building on this, we introduce (i) a segment-wise aggregation rule that averages only over clients contributing to each rank segment, avoiding dilution from zero-padded lower-rank clients, and (ii) a prefix-nested training strategy that optimizes each adapter under multiple rank truncations, encouraging useful signal to concentrate in low-rank prefix dimensions. Together, these components encourage a consistent low-rank prefix capturing the most task-relevant information, while higher-rank dimensions learn additional capacity. This allows low-rank clients to benefit from richer information contributed by higher-rank clients, as prefix dimensions are consistently learned and aggregated. Experiments demonstrate that our method consistently outperforms prior heterogeneous federated LoRA methods in accuracy and ROUGE-L, while achieving lower or comparable perplexity across multiple base models.
