Search papers, labs, and topics across Lattice.
This paper introduces MLLMCLIP, a novel framework that distills knowledge from a generative Multimodal Large Language Model (MLLM) into a discriminative CLIP model, effectively enhancing its compositionality without relying on synthetic data. By employing an attention-based per-layer token selection and a CKA-based distillation loss, MLLMCLIP addresses the architectural mismatch between the teacher and student models. The results show that this approach not only improves compositional accuracy but also yields significant gains in zero-shot classification and image-text retrieval tasks, outperforming previous methods.
Bypassing synthetic data, MLLMCLIP achieves state-of-the-art compositional accuracy in vision-language tasks through innovative feature-level distillation.
Pretrained vision-language models such as CLIP excel at zero-shot recognition but often fail at compositionality, particularly attribute-object and relational structures. Recent studies mitigate this issue by augmenting training with synthetic hard negatives generated by a cascade of large language models and text-to-image models, which incurs substantial pipeline overhead. We instead propose MLLMCLIP, a heterogeneous distillation framework that transfers multimodal knowledge directly from a generative Multimodal Large Language Model (MLLM) teacher into a discriminative CLIP student, bypassing synthetic data entirely. To bridge the architectural mismatch between the two paradigms, we introduce an attention-based per-layer token selection and a CKA-based distillation loss. Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.
