Search papers, labs, and topics across Lattice.
This paper investigates the generalization mechanisms of Parameter-Efficient Fine-Tuning (PEFT) methods, revealing that a small subset of sharp dimensions plays a critical role in their performance. By focusing on these dimensions, the authors introduce Flatness Preference Optimization (FlatPO), which optimizes the sharpness of these key dimensions to enhance generalization across various PEFTs. Experimental results confirm that FlatPO significantly improves performance, suggesting a more efficient approach to fine-tuning large multimodal models.
A small fraction of sharp dimensions governs the generalization of PEFT methods, and optimizing these can lead to substantial performance gains.
Parameter-Efficient Fine-Tuning (PEFT) methods provide a streamlined and efficient tool for adapting large models to domain-specific multimodal downstream tasks. Although these methods proved their tangible effects in practice, their principal aspects remain under-explored. Therefore we remain curious about the underlying generalization mechanisms in various PEFT methods and how they can be further enhanced. In this paper, we reveal the flatness preference widely present in various PEFTs, where a small fraction of sharp dimensions dominates the generalization of PEFT. This finding suggests an appealing possibility: we may be satisfied with a better generalization by merely attending to this small fraction of sharp dimensions instead of all of them. Furthermore, we propose Flatness Preference Optimization (FlatPO) to flatten these key sharpness dimensions, leading various PEFTs toward better generalization. Extensive experiments demonstrate the effectiveness of our findings and the proposed method. Code is available at https://github.com/Can-Lin/FlatPO.
