Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Text Prompt Boosting (TPB), an AdaBoost-inspired framework that enhances the performance of Vision-Language Models (VLMs) by focusing on misclassified examples during the construction of text prompts. By treating each text-prompt-based classifier as a weak learner, TPB aggregates their outputs into a robust ensemble, significantly improving classification accuracy across eleven benchmarks. The results demonstrate that TPB not only enhances the source model's performance but also maintains gains when applied to larger VLMs, addressing limitations of existing few-shot prompting methods.
Misclassified examples can be the key to unlocking significant performance improvements in Vision-Language Models through targeted text prompt boosting.
The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts. Handcrafted templates and Large Language Model (LLM)-generated descriptions not only make predictions more interpretable, but also enable reuse of the same prompts across heterogeneous VLMs. Recent works construct task-adapted text prompts with a small number of labeled images. However, existing few-shot text prompting methods do not explicitly focus on misclassified examples during prompt construction, leading to only marginal improvements even as more shots become available. To fully exploit few-shot supervision, we propose Text Prompt Boosting (TPB), an AdaBoost-inspired framework that treats each text-prompt-based classifier as a weak learner and sequentially aggregates them into a strong ensemble by explicitly targeting hard, misclassified examples. Extensive experiments show that TPB preserves task-intrinsic, model-agnostic cues in text space, enabling robust cross-model transfer. Across eleven classification benchmarks, TPB improves accuracy on the source model and preserves shot-driven gains when transferred to larger, more capable VLMs, where existing methods struggle to sustain such improvements.