Search papers, labs, and topics across Lattice.
This study evaluates five parameter-efficient fine-tuning (PEFT) methods for vision models on consumer-grade GPUs, focusing on their performance under strict memory constraints. By implementing an adaptive checkpointing strategy, the authors achieve significant reductions in peak memory usage while maintaining competitive accuracy and energy efficiency. Key findings reveal that QLoRA and BitFit can reduce energy consumption by 20-30% with minimal accuracy trade-offs, while DINOv2 outperforms traditional fine-tuned models in energy efficiency on CIFAR-100.
QLoRA and BitFit deliver substantial energy savings for fine-tuning vision models without sacrificing accuracy, challenging the notion that more resources always yield better performance.
Modern pretrained vision models achieve strong accuracy but demand substantial GPU memory for fine-tuning, making edge deployment impractical. This paper compares five parameter-efficient fine-tuning (PEFT) methods (Full FT, LoRA, AdaLoRA, QLoRA, BitFit) on Transformers- (ViT-Small, TinyViT) and Mamba-based vision backbones (Vim-Small, MambaVision-T) under an on-device VRAM budget (e.g., 2 GB), together with three gradient-checkpointing strategies (none, static, and a proposed memory-budget-aware adaptive algorithm); and we evaluate three families of foundation-model baselines: zero-shot contrastive vision language models (OpenCLIP, SigLIP), self-supervised vision backbones with lightweight evaluation protocols (DINOv2), and autoregressive VLMs for prompt-based classification (PaliGemma, MobileVLM, SmolVLM). Experiments on CIFAR-100 and DTD report accuracy, training time, energy, and the NetScore family of multi-objective metrics, which we extend with two deployment-aware variants. QLoRA and BitFit cut energy 20-30% at a 1-2% accuracy cost; the adaptive algorithm reduces peak memory 43-79% with 9-30% energy overhead. DINOv2 surpasses fine-tuned models on CIFAR-100 (0.917 vs. 0.897) at a fraction of the energy, while small autoregressive VLMs remain uncompetitive.
