Search papers, labs, and topics across Lattice.
This paper conducts a comparative study of various vision-language model (VLM) adaptation strategies for federated learning (FL) in remote sensing image classification, focusing on full fine-tuning, encoder-specific fine-tuning, prompt learning, and low-rank adaptation (LoRA). The research addresses the challenges posed by non-IID data across decentralized clients, revealing distinct trade-offs between generalization capability, communication overhead, and computational complexity. Key findings suggest that the choice of adaptation strategy significantly impacts model performance and efficiency, leading to practical guidelines for selecting VLM approaches in FL under varying constraints.
Selecting the right VLM adaptation strategy can drastically improve performance in federated learning for remote sensing, balancing efficiency and generalization.
Federated learning (FL) enables collaborative training of deep learning models across decentralized image archives without requiring data centralization. This paradigm is particularly relevant in remote sensing (RS), where legal regulations, privacy concerns, and bandwidth constraints restrict data sharing. However, the presence of training data heterogeneity across clients (known as non-IID data) can impede convergence and limit the generalization capability of the aggregated global model. To mitigate the adverse effects of training data heterogeneity, vision-language models (VLMs) can be leveraged in FL due to their transferable representations, which have demonstrated robustness under distribution shifts. However, their large parameter size may substantially increase communication overhead and local computational complexity in federated settings. Therefore, it is crucial to select an appropriate VLM adaptation strategy that balances the generalization ability with the communication and computational constraints. To address this issue, in this paper, we present the first comparative study of VLM adaptation strategies for FL in the context of RS image classification. We investigate full fine-tuning, encoder-specific fine-tuning, prompt learning, and low-rank adaptation (LoRA) tuning, and analyze them with respect to three criteria: 1) generalization capability under non-IID data, 2) communication overhead, and 3) local computational complexity. Experiments on BigEarthNet-S2, EuroSAT, RESISC45, and ImageNet reveal distinct trade-offs between task specialization, cross-domain generalization, and efficiency. Based on our findings, we derive a guideline for the selection of an appropriate VLM adaptation strategy in FL for RS image classification under different operational constraints. The code of this work is publicly available at https://git.tu-berlin.de/rsim/FL-RS-VLM.
