Search papers, labs, and topics across Lattice.
This paper benchmarks Large Multimodal Models (LMMs) against CLIP-like Vision-Language Models (VLMs) for closed-world classification, finding that LMMs, when using in-context learning, can match or exceed the performance of VLMs with cache-based adapters. The study extends to open-world classification, revealing that LMMs struggle with imperfect context, which motivates the introduction of CIRCLE, a training-free method for iteratively refining pseudo-labels for in-context examples. Experiments demonstrate that CIRCLE improves LMM performance in open-world classification, establishing a strong baseline compared to VLMs.
Forget specialized classifiers—LMMs, enhanced with in-context learning and a novel iterative refinement method (CIRCLE), can outperform even fine-tuned VLMs in both closed and open-world classification.
Which multimodal model should we use for classification? Previous studies suggest that the answer lies in CLIP-like contrastive Vision-Language Models (VLMs), due to their remarkable performance in zero-shot classification. In contrast, Large Multimodal Models (LMM) are more suitable for complex tasks. In this work, we argue that this answer overlooks an important capability of LMMs: in-context learning. We benchmark state-of-the-art LMMs on diverse datasets for closed-world classification and find that, although their zero-shot performance is lower than CLIP's, LMMs with a few in-context examples can match or even surpass contrastive VLMs with cache-based adapters, their"in-context"equivalent. We extend this analysis to the open-world setting, where the generative nature of LMMs makes them more suitable for the task. In this challenging scenario, LMMs struggle whenever provided with imperfect context information. To address this issue, we propose CIRCLE, a simple training-free method that assigns pseudo-labels to in-context examples, iteratively refining them with the available context itself. Through extensive experiments, we show that CIRCLE establishes a robust baseline for open-world classification, surpassing VLM counterparts and highlighting the potential of LMMs to serve as unified classifiers, and a flexible alternative to specialized models.
