Search papers, labs, and topics across Lattice.
This paper introduces MultiModal Code-Switching (MMCS), a pretraining paradigm that enhances the alignment between visual objects and language by interleaving them, thereby addressing the referential ambiguity found in traditional image-text pair models. By providing explicit object-level supervision, MMCS allows for improved semantic grounding and data efficiency, achieving comparable or superior performance with significantly fewer training samples. The results demonstrate that MMCS not only enhances visual grounding but also improves perception capabilities across different model sizes, indicating a robust method for multimodal learning.
MMCS achieves superior visual grounding with just 50K samples, outperforming traditional models trained on 600K image-text pairs.
Existing Multimodal Large Language Models (MLLMs) predominantly rely on image-text pairs for modality alignment pretraining, mapping global image representations to long textual descriptions. However, this image-level alignment suffers from referential ambiguity: models struggle to infer the correspondences between multiple visual objects and textual entities from the global representation, leading to data inefficiency and suboptimal semantic grounding. To address this, we propose MultiModal Code-Switching (MMCS), a novel pretraining paradigm that provides explicit object-level supervision. Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding. We further develop a scalable data synthesis pipeline to generate a pretraining dataset of 773K samples with accurate object-entity correspondences. Experiments show that MMCS is highly data-efficient: with only 50K samples, it matches or surpasses models trained on 600K image-text pairs. Furthermore, MMCS consistently improves visual grounding and perception capabilities across varying model scales.
