Search papers, labs, and topics across Lattice.
This paper introduces WikiCLIP, a contrastive learning framework for open-domain visual entity recognition (VER) that uses large language model embeddings for entity representations. To improve alignment between text and images, they propose a Vision-Guided Knowledge Adaptor (VGKA) that aligns textual semantics with visual cues at the patch level, and a Hard Negative Synthesis Mechanism to generate challenging negative samples. Experiments on OVEN demonstrate that WikiCLIP achieves a 16% improvement on the unseen set and reduces inference latency by nearly 100 times compared to generative models like AutoVER.
Forget slow generative models: WikiCLIP delivers a 16% accuracy boost in visual entity recognition with 100x faster inference by cleverly combining CLIP-style contrastive learning with vision-guided knowledge adaptation.
Open-domain visual entity recognition (VER) seeks to associate images with entities in encyclopedic knowledge bases such as Wikipedia. Recent generative methods tailored for VER demonstrate strong performance but incur high computational costs, limiting their scalability and practical deployment. In this work, we revisit the contrastive paradigm for VER and introduce WikiCLIP, a simple yet effective framework that establishes a strong and efficient baseline for open-domain VER. WikiCLIP leverages large language model embeddings as knowledge-rich entity representations and enhances them with a Vision-Guided Knowledge Adaptor (VGKA) that aligns textual semantics with visual cues at the patch level. To further encourage fine-grained discrimination, a Hard Negative Synthesis Mechanism generates visually similar but semantically distinct negatives during training. Experimental results on popular open-domain VER benchmarks, such as OVEN, demonstrate that WikiCLIP significantly outperforms strong baselines. Specifically, WikiCLIP achieves a 16% improvement on the challenging OVEN unseen set, while reducing inference latency by nearly 100 times compared with the leading generative model, AutoVER. The project page is available at https://artanic30.github.io/project_pages/WikiCLIP/
