Search papers, labs, and topics across Lattice.
This paper introduces the Multi-Modal Quantum Adapter (MQAdapter), a novel approach for fine-tuning Vision-Language Models (VLMs) that enhances few-shot learning by addressing the challenge of fine-grained discrimination among visually similar categories. By leveraging quantum computation, MQAdapter retrieves the Top-K category candidates as semantic anchors and refines visual features through a cross-modal quantum learning mechanism, enabling more effective modeling of higher-order interactions. Evaluations across 15 datasets reveal that MQAdapter achieves superior performance with fewer trainable parameters compared to traditional methods, highlighting its efficiency and effectiveness in fine-tuning VLMs.
Quantum computation can significantly enhance fine-grained discrimination in Vision-Language Models, achieving better performance with fewer parameters.
Large-scale Vision-Language Models have demonstrated impressive transfer learning capabilities across a wide range of tasks. For few-shot classification, we observe that VLMs exhibit a notable ability to filter candidate categories and thus achieve high Top-K accuracy. However, they often struggle with fine-grained discrimination among visually similar categories, resulting in unsatisfactory Top-1 performance, as shown in Figure 1. Existing studies on VLM adapters generally focus on global alignment between visual and textual representations in the feature space, but fail to exploit semantically similar categories to refine fine-grained visual representations. Based on these observations, we propose a novel coarse-to-fine VLM fine-tuning approach for few-shot learning that leverages quantum computation, termed the Multi-Modal Quantum Adapter (MQAdapter). Specifically, MQAdapter first retrieves the Top-K category candidates most similar to the input image and uses them as semantic anchors. It then employs a cross-modal quantum learning mechanism to refine visual features under the guidance of these anchors. The core of this mechanism is the encoding of visual and textual features into quantum states. By leveraging quantum entanglement and superposition in a high-dimensional Hilbert space, MQAdapter effectively models higher-order cross-modal interactions, producing more discriminative representations than traditional Euclidean adapters. MQAdapter is parameter-efficient and can be integrated with various existing fine-tuning algorithms to achieve further performance gains. Evaluations on 15 datasets demonstrate the effectiveness of MQAdapter while requiring fewer trainable parameters.
