Search papers, labs, and topics across Lattice.
This paper formalizes cross-modal coreference as a key challenge for Omni-LLMs, requiring models to identify shared referents across modalities. They introduce CrossOmni, a dataset with nine tasks and human-designed rationales, to evaluate this capability in 13 Omni-LLMs. Results show systematic weaknesses, which the authors address via a training-free In-Context Learning method and a training-based SFT+GRPO framework, both yielding significant performance gains and generalization.
Omni-LLMs struggle to identify the same objects across different modalities, but a new dataset and training strategies can significantly improve their cross-modal reasoning.
Omni Large Language Models (Omni-LLMs) have demonstrated impressive capabilities in holistic multi-modal perception, yet they consistently falter in complex scenarios requiring synergistic omni-modal reasoning. Beyond understanding global multimodal context, effective reasoning also hinges on fine-grained cross-modal alignment, especially identifying shared referents across modalities, yet this aspect has been largely overlooked. To bridge this gap, we formalize the challenge as a cross-modal coreference problem, where a model must localize a referent in a source modality and re-identify it in a target modality. Building on this paradigm, we introduce CrossOmni, a dataset comprising nine tasks equipped with human-designed reasoning rationales to evaluate and enhance this capability. Experiments on 13 Omni-LLMs reveal systematic weaknesses in cross-modal coreference, which we attribute to the absence of coreference-aware thinking patterns. To address this, we enhance cross-modal alignment via two strategies: a training-free In-Context Learning method and a training-based SFT+GRPO framework designed to induce such thinking patterns. Both approaches yield substantial performance gains and generalize effectively to collaborative reasoning tasks. Overall, our findings highlight cross-modal coreference as a crucial missing piece for advancing robust omni-modal reasoning.
