Search papers, labs, and topics across Lattice.
This paper introduces a Context-Enhanced Multimodal Knowledge Graph (CEMMKG) framework that integrates both local and global textual contexts to improve the performance of multimodal retrieval-augmented generation (RAG) systems. By addressing the semantic gap between images and text, the CEMMKG enriches visual data with semantically relevant textual information, enhancing the multimodal GraphRAG's ability to generate accurate outputs. Experimental results on vision-centric datasets show that CEMMKG significantly boosts performance across various MMKG-based RAG methods, highlighting its versatility and effectiveness.
Bridging the semantic gap between images and text can dramatically enhance multimodal RAG performance, as shown by our novel Context-Enhanced MMKG framework.
Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models (LLMs) and multimodal large language models (MLLMs). In particular, knowledge graph (KG)-based RAG leverages structured knowledge to provide (M)LLMs with high-quality external information. Building on these works, recent studies have explored multimodal knowledge graphs (MMKGs) as knowledge bases for GraphRAG. This enables Graph RAG to integrate knowledge across multiple modalities, thereby further enhancing its performance. However, existing MMKG-based RAG methods generally follow a common pipeline in which different modalities are largely processed independently before being fusion. As a result, textual context is only used to a limited extent during visual information extraction and subsequent multimodal knowledge fusion. This brings a semantic gap between images and text which limits the multimodal GraphRAG performance. To address this issue, we propose a novel framework for constructing a Context-Enhanced MMKG (CEMMKG) to better support multimodal GraphRAG. The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes. Local context goes beyond the surrounding text by incorporating sentences that are semantically related to the image, while global context provides a summary of the entire passage. We further introduce a multi-granularity design for the local context, allowing it to capture semantically relevant information at different levels of detail. Extensive experiments on the selected vision-centric dataset validate that CEMMKG is effective in leveraging contextual information to improve MMKG-based RAG performance. Moreover, its effectiveness across different MMKG-based RAG methods demonstrates its broad applicability.
