Search papers, labs, and topics across Lattice.
This paper reformulates vision-language grounding as a bidirectional concept correspondence problem, addressing the limitations of traditional unidirectional localization approaches. By introducing the ConCor-1 model, which employs learnable bridge tokens to predict text-image correspondences without pre-specified text spans, the authors unify various grounding tasks into a single framework. Experimental results demonstrate significant improvements in correspondence F1 scores, with a 48% increase on long-caption datasets and a 29% increase in zero-shot LVIS scenarios, highlighting the effectiveness of the proposed method.
Grounding tasks can be unified into a single correspondence prediction problem, leading to a 48% boost in performance on complex datasets.
Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as $\textit{bidirectional concept correspondence}$ over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce $\textbf{ConCor-1}$, a grounding model built on top of a pretrained vision-language model. It uses learnable $\textit{bridge tokens}$ to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that $\textbf{ConCor-1}$ consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.
