Search papers, labs, and topics across Lattice.
This paper introduces Low-Rank Convolutional Adaptation (LoCA), a novel framework designed to enhance Parameter-Efficient Fine-Tuning (PEFT) of Vision Foundation Models (VFMs) by addressing the spatial-channel entanglement inherent in convolutional layers. By decoupling channel and spatial adaptation using low-rank channel adaptation and refining spatial bases through Singular Value Decomposition (SVD), LoCA preserves the spatial topology of convolutional kernels while improving adaptation efficiency. Experimental results demonstrate that LoCA not only maintains pre-trained spatial priors but also achieves competitive or state-of-the-art performance across various tasks, including fine-grained classification and domain-generalized semantic segmentation.
LoCA achieves state-of-the-art performance in vision tasks while preserving spatial priors, revolutionizing how we adapt convolutional models without full fine-tuning.
Pre-trained Vision Foundation Models (VFMs) provide strong visual representations for diverse downstream tasks. The key challenge of VFM adaptation stems from the prohibitive costs of full fine-tuning and catastrophic forgetting. To address this, Low-Rank Adaptation (LoRA) has emerged as the prevailing paradigm for Parameter-Efficient Fine-Tuning (PEFT). However, LoRA is typically designed for transformer self-attention layers parameterized by 2D matrices. Since convolutional kernels inherently couple spatial and channel information within a 4D tensor, forcing them into a monolithic 2D matrix disrupts the inherent spatial topology. In this paper, we propose Low-Rank Convolutional Adaptation (LoCA), a convolution-aware PEFT framework that addresses spatial-channel entanglement by decoupling channel and spatial adaptation. LoCA introduces a low-rank channel adaptation for dense cross-channel mixing and refines spatial bases extracted from pre-trained kernels via Singular Value Decomposition (SVD). Experimental results show that LoCA preserves pre-trained spatial priors and achieves competitive or state-of-the-art performance across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks.
