Search papers, labs, and topics across Lattice.
This paper introduces FeDiSyn, a unified framework that optimally integrates synthetic-image pre-training with federated fine-tuning (FedFT) for large vision models (LVMs) to address challenges such as resource constraints and non-IID data. By employing a scaling law to determine the optimal number of synthetic images and utilizing diffusion-based generation to capture device-specific features, the framework enhances training efficiency and reduces communication costs. Experimental results indicate that FeDiSyn can decrease completion time by over 52.5% and communication costs by over 97.2%, while maintaining accuracy comparable to existing state-of-the-art methods.
Reducing federated fine-tuning time by over 52% while slashing communication costs by more than 97% could revolutionize how we deploy large vision models on resource-constrained devices.
Federated fine-tuning (FedFT) enables adapting pre-trained large vision models (LVMs) on distributed, privacy-sensitive devices, while its practical deployment is hindered by three critical challenges: resource constraints, system heterogeneity, and non-IID data. While prior studies partially address these issues, e.g., by pre-training initial models on synthetic images to mitigate the adverse effects of non-IID data, or leveraging parameter-efficient fine-tuning (PEFT) methods like low-rank adaptation (LoRA) to reduce resource consumption, they remain inadequate and fragmented. Specifically, existing synthetic image generation methods fail to capture device-specific feature distributions, while current PEFT-based FedFT methods often undervalue weaker devices that may provide critical information. More importantly, the separate optimization of pre-training and FedFT neglects their inherent connection, lacking a holistic perspective to maximize training efficiency. To overcome these limitations, we propose FeDiSyn, a unified framework that holistically considers the interplay between pre-training and FedFT to minimize the overall LVM training time. Specifically, FeDiSyn introduces: (i) a scaling law for FedFT pre-training to determine the optimal number of synthetic images, balancing pre-training benefit against generation/pre-training cost, (ii) diffusion-based synthetic image generation that captures device-specific feature distributions for pre-training to tackle non-IID data, and (iii) a contribution-aware LoRA configuration and bandwidth allocation algorithm for FedFT to ensure that informative devices are effectively utilized while addressing system heterogeneity. Experimental results on the real-world testbed demonstrate that FeDiSyn reduces completion time by over 52.5% and communication cost by over 97.2%, while achieving comparable accuracy to state-of-the-art solutions.
