Search papers, labs, and topics across Lattice.
The paper introduces Multi-Modal Interactive Agent Layer (MAIL) and its extension MAIL++, parameter-efficient fine-tuning (PEFT) paradigms that directly embed cross-modal coupling into the intrinsic computation modules of VLMs by inserting lightweight agent layers after core modules. MAIL++ enhances this by enabling bidirectional cross-modal exchange through meta agent layers and bridges. Experiments on few-shot image classification and cross-domain retrieval show MAIL and MAIL++ outperform state-of-the-art PEFT methods while maintaining computational efficiency by re-parameterizing agent layers into the frozen backbone at inference.
Forget clunky external modules: MAIL++ injects cross-modal learning directly into the VLM's core, boosting few-shot performance without sacrificing inference speed.
Adapting large vision-language models (VLMs) such as CLIP to downstream tasks remains challenging, as full fine-tuning is computationally prohibitive and prone to overfitting in low-data regimes. Parameter-efficient fine-tuning (PEFT) alleviates these issues with lightweight prompt- or adapter-based modules, and cross-modal coupling has proven especially effective by strengthening interactions between vision and language. However, existing coupling mechanisms predominantly rely on external auxiliary modules, leading to indirect, coarse-grained interactions that are structurally decoupled from the original VLM and thus limit representational expressiveness. In this paper, we propose Multi-Modal Interactive Agent Layer (MAIL), a PEFT paradigm that embeds cross-modal coupling directly into the intrinsic computation modules of VLMs. MAIL freezes the backbone and inserts lightweight agent layers after core modules, such as LayerNorm, to approximate the parameter updates induced by full fine-tuning. To couple visual and textual streams at this level, we introduce a bottleneck-based text-to-image bridge that jointly optimizes paired agent layers across modalities, coordinating the adaptation of corresponding computation modules. We further present MAIL++, which enables bidirectional cross-modal exchange through a meta agent layer, a meta-text bridge, and a meta-image bridge. At inference time, all agent layers are re-parameterized into the frozen backbone, preserving the original computational efficiency. Extensive experiments on few-shot image classification and few-shot universal cross-domain retrieval demonstrate that MAIL and MAIL++ consistently outperform state-of-the-art PEFT methods.
