Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces HeBA (Heterogeneous Bottleneck Adapter), a novel adapter architecture for VLMs that incorporates modality-specific inductive biases by processing visual tokens with 2D depthwise-separable convolutions and text tokens with dense linear projections. HeBA also employs a compression bottleneck and Kaiming initialization to learn compact, robust features and accelerate convergence. Experiments on 11 few-shot benchmarks demonstrate that HeBA achieves state-of-the-art performance and improved stability compared to standard adapter approaches.
VLMs can be made more robust and accurate by ditching "one-size-fits-all" adapters and embracing architectures tailored to the unique structure of visual and textual data.
Adapting large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks often suffers from a"one-size-fits-all"architectural approach, where visual and textual tokens are processed uniformly by wide, generic adapters. We argue that this homogeneity ignores the distinct structural nature of the modalities -- spatial locality in images versus semantic density in text. To address this, we propose HeBA (Heterogeneous Bottleneck Adapter), a unified architectural framework that introduces modality-specific structural inductive biases. HeBA departs from conventional designs through three key architectural innovations: (1) Heterogeneity: It processes visual tokens via 2D depthwise-separable convolutions to preserve spatial correlations, while distinctively processing text tokens via dense linear projections to capture semantic relationships; (2) Bottleneck Regularization: Unlike standard expanding adapters, HeBA employs a compression bottleneck (D ->D/4) that explicitly forces the model to learn compact, robust features and acts as a structural regularizer; and (3) Active Gradient Initialization: We challenge the restrictive zero-initialization paradigm, utilizing a Kaiming initialization strategy that ensures sufficient initial gradient flow to accelerate convergence without compromising the frozen backbone's pre-trained knowledge. Extensive experiments demonstrate that HeBA's architecturally specialized design achieves superior stability and accuracy, establishing a new state-of-the-art on 11 few-shot benchmarks. Code is available at https://github.com/Jahid12012021/VLM-HeBA.