Search papers, labs, and topics across Lattice.
The paper introduces a plug-and-play module to improve vision language models' (VLMs) reasoning on rare objects without finetuning, addressing the challenge of limited rare object instances in pretraining data. The module learns multi-modal class embeddings for rare objects using vision foundation models and synonym-augmented text descriptions. These embeddings refine visual tokens via an attention-based enhancement and generate informative hints injected into text prompts, guiding the VLM's attention.
Boost a VLM's rare-object reasoning without any finetuning by injecting learned embeddings from vision foundation models and synonym-augmented text.
Vision language models (VLMs) have achieved remarkable success in broad visual understanding, yet they remain challenged by object-centric reasoning on rare objects due to the scarcity of such instances in pretraining data. While prior efforts alleviate this issue by retrieving additional data or introducing stronger vision encoders, these methods are still computationally intensive during finetuning VLMs and don't fully exploit the original training data. In this paper, we introduce an efficient plug-and-play module that substantially improves VLMs' reasoning over rare objects by refining visual tokens and enriching input text prompts, without VLMs finetuning. Specifically, we propose to learn multi-modal class embeddings for rare objects by leveraging prior knowledge from vision foundation models and synonym-augmented text descriptions, compensating for limited training examples. These embeddings refine the visual tokens in VLMs through a lightweight attention-based enhancement module that improves fine-grained object details. In addition, we use the learned embeddings as object-aware detectors to generate informative hints, which are injected into the text prompts to help guide the VLM's attention toward relevant image regions. Experiments on two benchmarks show consistent and substantial gains for pretrained VLMs in rare object recognition and reasoning. Further analysis reveals how our method strengthens the VLM's ability to focus on and reason about rare objects.
