Search papers, labs, and topics across Lattice.
This study compares the performance of Google's Gemini Embedding 2, a natively multimodal embedding model, against frontier large language models (LLMs) like GPT-4.1 and Claude Sonnet 4.6 in the context of zero-shot text-to-image retrieval on the Flickr30k dataset. The results reveal that the LLMs can match the performance of the native multimodal model, challenging the assumption that specialized multimodal systems are superior. Furthermore, the research highlights that precomputed multimodal embeddings offer advantages for low-latency applications, suggesting a shift in preference for certain use cases.
LLMs can rival specialized multimodal embeddings in zero-shot retrieval tasks, reshaping the landscape of multimodal AI applications.
Multimodal retrieval and classification across different types of media, spanning text, images,video and audio, has traditionally relied on dual-encoder models that align visual and textual representations through contrastive learning. The March 2026 release of Gemini Embedding 2, Google's first natively multimodal embedding model to map text, images, video, audio, and documents into a single shared space, raises competition among multimodal retrieval systems. Simultaneously, frontier Large language models (LLMs) have also demonstrated strong visual understanding, raising the question of whether they can serve as effective zero-shot rankers. Our study provides the first direct comparison of native multimodal embeddings against LLM-based visual ranking on Flickr30k. We observe that GPT-4.1 and Claude Sonnet 4.6 perform on par with Gemini Embedding 2. Additionally, once embeddings are precomputed, multimodal embeddings are better suited for low-latency applications.
