Search papers, labs, and topics across Lattice.
This paper introduces AGE, an adaptive-masking technique designed to enhance graph embeddings for retrieval-augmented generation (GraphRAG) in large language models (LLMs). By addressing the misalignment between graph-based and text-based latent features through a self-supervised learning approach, AGE effectively predicts non-key nodes, which improves the efficiency of the embedding process. Experimental results demonstrate that AGE significantly boosts accuracy in GraphQA tasks across four diverse benchmark datasets, outperforming existing methods that utilize non-parametric search components.
Achieving superior accuracy in GraphQA tasks, AGE reveals that focusing on non-key nodes can transform graph embedding efficiency for LLMs.
GraphRAG is an extension of retrieval-augmented generation (RAG) that supports large language models (LLMs) by referring to graph-structured data as external knowledge. While this technique ideally captures intricate relationships, it often struggles with graph representations for LLMs, particularly for frozen LLMs, due to the misalignment between graph-based and text-based latent features. We tackle this issue by introducing the {\it Adaptive-masking for Graph Embedding (AGE)}. AGE employs a Transformer in a mask-based self-supervised learning (SSL) approach. We designed the architecture similar to text embedding encoders, addressing the latent feature misalignment. In contrast to natural language texts, graphs are concise representations, and there exist {\it key nodes} that hold dominant contextual information, which are challenging to predict from their surroundings. Masking such key nodes leads to inefficiency in the SSL process. Therefore, AGE focuses on predicting nodes apart from key nodes, utilizing a learnable node sampler. Our experimental results indicate that AGE significantly improves approaches using non-parametric search component in GraphQA tasks, achieving superior accuracy across four benchmark datasets with distinct characteristics.
