Search papers, labs, and topics across Lattice.
This paper introduces FACT, a novel three-phase hierarchical finetuning framework designed to enhance the performance of pretrained models in active learning scenarios by addressing the issues of distribution shift and overfitting. By systematically exploring finetuning methodologies and employing frozen feature augmentation strategies, FACT significantly improves model robustness and generalization across various image classification datasets. The results reveal that under low sampling ratios, FACT achieves over 20% performance gains on the ViT model across benchmarks like CIFAR10, CIFAR100, and ImageNet-1k, establishing new state-of-the-art results while ensuring parameter efficiency.
Achieving over 20% performance gains under low sampling ratios, FACT redefines the landscape of active finetuning for pretrained models.
The main goal of active finetuning is to improve a pretrained model's performance on a specific task or domain by finetuning it with carefully selected informative or challenging data. Previous research has predominantly focused on the active aspect (i.e., data selection) while uniformly employing full finetuning for model adaptation, which inevitably distorts pretrained features due to distribution shift. This issue becomes particularly pronounced when the model size is large relative to the finetuning data quantity, leading to heightened overfitting risks. To address this critical gap, we formally outline the FiAF task that emphasizes systematic exploration of finetuning methodologies in active learning. We propose FACT, a three-phase hierarchical finetuning framework featuring both efficiency and simplicity, specifically designed for active finetuning scenarios. Our comprehensive experiments span: (1) Three major dataset categories encompassing classic (CIFAR10, CIFAR100, ImageNet-1k), imbalanced (CIFAR10-LT, CIFAR100-LT), and fine-grained (StanfordCars, FGVCAircraft) image classification datasets, each evaluated under 3-5 distinct sampling ratios; (2) Diverse pretrained architectures including Convolutional Neural Network (ConvNeXt), Vision Transformer (ViT), and Vision LSTM (ViL) networks; (3) A systematic investigation of frozen feature augmentation (FroFA) strategies. (4) A comprehensive and rigorous analysis of efficiency and generalizability. The results demonstrate significant improvements with strong generalization and robustness. Notably, under low sampling ratios, our framework achieves remarkable performance gains of over 20% on the ViT model for CIFAR10, CIFAR100, and ImageNet-1k benchmarks. This systematic approach establishes new state-of-the-art performance while maintaining parameter efficiency, proving particularly effective when labeled data is scarce.
