Search papers, labs, and topics across Lattice.
This paper addresses the underexplored area of visual identity discrimination within Universal Multimodal Embeddings (UMEs), which is essential for tasks like instance retrieval and identity preservation. The authors introduce a unified formulation for visual identity discrimination (VisID) and present the Multimodal Visual Identity Embedding Benchmark (MVEB), a comprehensive dataset for evaluating and training these embeddings. Their proposed learning framework effectively optimizes both general multimodal and visual identity representations, achieving strong identity discrimination while maintaining competitive performance across multimodal tasks.
Visual identity discrimination is the missing link in Universal Multimodal Embeddings, and this work provides a robust framework to integrate it seamlessly.
Universal Multimodal Embeddings (UMEs) aim to unify various modalities and tasks into a shared representation space. In recent years, this field has witnessed substantial progress driven by the development of Multimodal Large Language Models (MLLMs). However, a crucial capability, visual identity discrimination, remains underexplored in existing UME methods, despite its critical role in a wide range of tasks, including instance retrieval, re-identification, and identity preservation in AI-generated content. To bridge this gap, we propose a unified formulation for visual identity discrimination~(VisID) and introduce $\textbf{MVEB}$ ($\textbf{M}$ultimodal $\textbf{V}$isual Identity $\textbf{E}$mbedding $\textbf{B}$enchmark), a large-scale benchmark curated from both real-world and synthetic datasets to support evaluation and training. Furthermore, we present a simple yet effective learning framework that jointly optimizes general multimodal and visual identity representations through a carefully designed identity-aware sampling mechanism. Extensive experiments demonstrate that our approach successfully endows UMEs with strong identity discrimination capability and maintains competitive general multimodal performance. We believe this work not only illuminates a critical yet neglected capability, but also takes a step toward more holistic universal multimodal embeddings. Code and data are available at \href{https://chrisclear3.github.io/MVEB}{MVEB}.
DAMO