Search papers, labs, and topics across Lattice.
This paper introduces FisherAdapTune, a novel framework for parameter-efficient fine-tuning (PEFT) that dynamically selects parameter groups based on the temporal drift of their Fisher geometry, rather than relying on static architectural heuristics. By leveraging a PAC-Bayesian perspective, the authors decompose the generalization error bound into Fisher-weighted update costs, allowing them to freeze parameter groups with stabilized curvature contributions to enhance adaptation efficiency. Experimental results on a segmentation task demonstrate that FisherAdapTune significantly improves both in-distribution performance and zero-shot transfer, highlighting the effectiveness of using Fisher structural drift as a signal for adaptive fine-tuning.
Freezing parameter groups based on Fisher structural drift can enhance fine-tuning efficiency and boost performance in downstream tasks.
Parameter-efficient fine-tuning (PEFT) aims to adapt pretrained models with a small trainable parameter subset, however, most existing methods choose this subset from fixed architectural heuristics rather than using dynamic, task-aware criteria. We introduce \textbf{FisherAdapTune}, a Fisher-guided Adaptive Fine-Tuning framework that progressively selects parameter groups by tracking the temporal drift of their Fisher geometry. Starting from a PAC-Bayesian view of fine-tuning, we decompose the generalization error bound into Fisher-weighted update costs and show that parameter groups whose curvature contribution has stabilized can be frozen to reduce the error bound without interrupting the remaining adaptation dynamics. FisherAdapTune formulates this criterion with a scale-invariant Jensen-Shannon distance between consecutive Fisher distributions, yielding an adaptive active parameter set. We evaluate our approach on a downstream segmentation task, and results show FisherAdapTune improves the in-distribution performance and zero-shot transfer in multiple settings, validating that Fisher structural drift is a useful signal for efficient, task-aware adaptation. We release our \href{https://github.com/AtlasAnalyticsLab/FisherAdapTune}{code} publicly to enable further application of our proposed approach.
Mila