Search papers, labs, and topics across Lattice.
The paper addresses the challenge of suboptimal hyperparameter settings in parameter-efficient fine-tuning (PEFT) methods like LoRA and AdaptFormer when applied to domain-specific transfer learning tasks such as remote sensing. They propose MetaPEFT, a meta-learning approach that dynamically adjusts module influence during fine-tuning by learning adaptive scalers for module insertion, layer selection, and module-wise learning rates. Experiments on remote sensing and natural image datasets demonstrate that MetaPEFT achieves state-of-the-art performance, particularly improving tail-class accuracy with a small number of trainable parameters.
Forget grid search: MetaPEFT meta-learns the optimal hyperparameters for parameter-efficient fine-tuning, dynamically adapting module influence to boost performance on long-tailed data.
Training large foundation models from scratch for domain-specific applications is almost impossible due to data limits and long-tailed distributions -- taking remote sensing (RS) as an example. Fine-tuning natural image pre-trained models on RS images is a straightforward solution. To reduce computational costs and improve performance on tail classes, existing methods apply parameter-efficient fine-tuning (PEFT) techniques, such as LoRA and AdaptFormer. However, we observe that fixed hyperparameters -- such as intra-layer positions, layer depth, and scaling factors, can considerably hinder PEFT performance, as fine-tuning on RS images proves highly sensitive to these settings. To address this, we propose MetaPEFT, a method incorporating adaptive scalers that dynamically adjust module influence during fine-tuning. MetaPEFT dynamically adjusts three key factors of PEFT on RS images: module insertion, layer selection, and module-wise learning rates, which collectively control the influence of PEFT modules across the network. We conduct extensive experiments on three transfer-learning scenarios and five datasets in both RS and natural image domains. The results show that MetaPEFT achieves state-of-the-art performance in cross-spectral adaptation, requiring only a small amount of trainable parameters and improving tail-class accuracy significantly.
