Search papers, labs, and topics across Lattice.
This paper introduces a plug-and-adapt method for multimodal coreference resolution (MCR) that leverages a pre-trained alignment model to enhance performance without the need for extensive training on target datasets. By integrating visual and categorical cues through similarity aggregation and evidence theory, the approach achieves significant improvements in coreference resolution tasks. Experiments on the Coreference Image Narratives (CIN) benchmark show a 5.31% and 2.12% increase in CoNLL F1 scores compared to state-of-the-art methods and popular vision-language large models, respectively.
Achieving over 5% improvement in coreference resolution without the need for resource-intensive training on target datasets could revolutionize multimodal AI applications.
Visual information helps resolve ambiguity in coreference resolution, leading to notable performance gains. However, existing Multi-modal Coreference Resolution (MCR) methods require training with (partially) annotated data from the target dataset before they can be applied, preventing their direct usability and raising concerns about generalization. While Vision-Language Large Models (VLLMs) with billions of parameters offer promising zero-shot capabilities, they remain largely inaccessible. Their massive size limits deployability, and many are only accessible through paid APIs. In this paper, we propose a plug-and-adapt method that strategically adapts a carefully pre-trained \emph{alignment model} for immediate use in MCR tasks, designed to eliminate the need for training on scarce benchmark datasets or relying on resource-intensive VLLMs. Specifically, we first pre-train a fine-grained alignment model between textual and visual contextual information using vision-language alignment datasets. We then repurpose the alignment model to MCR through similarity aggregation by fusing visual and categorical cues with evidence theory, thereby enhancing effectiveness. Experiments on the Coreference Image Narratives (CIN) benchmark dataset demonstrate the effectiveness of our method, achieving a 5.31\% and 2.12\% improvement in CoNLL F1 over SOTA dedicated methods and popular VLLMs, respectively. We further evaluate our method on a masked CIN dataset for robustness testing and on a specially constructed VCR-MCR dataset for generalization assessment, with results confirming both capabilities.
