Search papers, labs, and topics across Lattice.
The paper introduces Mario, a framework for reasoning on multimodal graphs (MMGs) using LLMs, addressing challenges of weak cross-modal consistency and heterogeneous modality preference. Mario employs a graph-conditioned VLM that refines textual and visual features via cross-modal contrastive learning guided by graph topology. It also uses a modality-adaptive graph instruction tuning mechanism to present the most informative modality configuration to the LLM. Experiments on MMG benchmarks show Mario outperforms state-of-the-art graph models in node classification and link prediction in both supervised and zero-shot settings.
LLMs can now reason more effectively on multimodal graphs thanks to a new framework that aligns visual and textual features using graph structure and intelligently routes the most relevant modalities to the LLM.
Recent advances in large language models (LLMs) have opened new avenues for multimodal reasoning. Yet, most existing methods still rely on pretrained vision-language models (VLMs) to encode image-text pairs in isolation, ignoring the relational structure that real-world multimodal data naturally form. This motivates reasoning on multimodal graphs (MMGs), where each node has textual and visual attributes and edges provide structural cues. Enabling LLM-based reasoning on such heterogeneous multimodal signals while preserving graph topology introduces two key challenges: resolving weak cross-modal consistency and handling heterogeneous modality preference. To address this, we propose Mario, a unified framework that simultaneously resolves the two above challenges and enables effective LLM-based reasoning over MMGs. Mario consists of two innovative stages. Firstly, a graph-conditioned VLM design that jointly refines textual and visual features through fine-grained cross-modal contrastive learning guided by graph topology. Secondly, a modality-adaptive graph instruction tuning mechanism that organizes aligned multimodal features into graph-aware instruction views and employs a learnable router to surface, for each node and its neighborhood, the most informative modality configuration to the LLM. Extensive experiments across diverse MMG benchmarks demonstrate that Mario consistently outperforms state-of-the-art graph models in both supervised and zero-shot scenarios for node classification and link prediction. The code will be made available at https://github.com/sunyuanfu/Mario.
