Search papers, labs, and topics across Lattice.
This paper explores repurposing the classification heads of pre-trained vision models as semantic prototypes for vision-language alignment, eliminating the need for extensive paired data. By recycling these classification weights as semantic anchors and using them for data augmentation, the method achieves zero-shot alignment and improves performance in cross-modal tasks. Experiments show consistent accuracy gains when integrating this approach with existing post-hoc alignment techniques across various tasks.
Discarded classification weights from pre-trained vision models can be resurrected as powerful semantic prototypes, unlocking zero-shot vision-language alignment and boosting cross-modal performance.
Vision-Language Models (VLMs) excel at tasks like zero-shot classification and cross-modal retrieval by mapping images and text to a shared space, but this requires expensive end-to-end training with massive paired datasets. Current post-hoc alignment methods reduce computational costs by connecting pretrained encoders through lightweight mappings, yet still demand substantial paired data. In this work, we investigate the potential of repurposing the classification heads of pretrained vision models as semantic prototypes. The recycling of these weights, typically discarded after pretraining, unlocks two distinct capabilities: it enables zero-shot alignment by using weights as semantic anchors, and serves as a robust data augmentation strategy by mixing these prototypes with real image-text pairs. We demonstrate that integrating our approach with several state-of-the-art post-hoc alignment techniques consistently boosts accuracy in cross-modal retrieval, zero- and few-shot classification tasks.
