Search papers, labs, and topics across Lattice.
This paper introduces the CureLLM framework, which enhances graph-aware Large Language Models (LLMs) by incorporating edge-level information to improve the alignment between graph and textual representations. The authors provide a theoretical analysis demonstrating that traditional node-level alignment neglects crucial edge information, leading to suboptimal performance due to the over-squashing phenomenon. Experimental results across 11 real-world datasets show that CureLLM significantly outperforms 20 existing methods, validating the importance of edge-aware modeling in LLMs.
Neglecting edge information in graph-aware LLMs leads to significant performance bottlenecks, but the CureLLM framework unlocks superior alignment by leveraging curvature-enhanced graph representations.
Recently, graph-aware Large Language Models (LLMs) have shown promising capabilities in jointly modeling graph-structured data and textual information. Existing approaches typically employ a graph encoder and a frozen LLM to obtain node representations from graph and textual views, followed by node-level alignment to bridge the two modalities. However, such alignment mechanisms primarily focus on node information while overlooking edge-level structures, leading to suboptimal information propagation across views. In this work, we conduct a comprehensive theoretical analysis to uncover why node-level alignment is insufficient for aligning textual and graph representations. Specifically, we prove theoretically for the first time that neglecting edge information leads to suboptimal solutions and negatively curved edges induce bottlenecked information flow, giving rise to the over-squashing phenomenon between graph and textual views. To address the two challenges, we innovatively proposed a CureLLM framework of Curvature-enhanced Graph Representations for Large Language Model whose goal is to inject the signals of edge information into the existing LLMs. Specifically, CureLLM first introduces the training-free textual prompt mechanism to make the LLM model generate the output directly based on the edge-aware prompt without learnable parameter costs. Furthermore, a novel curvature-aware graph representation learning is designed to capture the edge structure information to enhance the downstream tasks, where the message passing between text and graph representations only depends on edges with positive curvature. Finally, we conduct evaluations with 20 different compared methods on 11 real world datasets from various domains and the experiment results demonstrate the superiority of our proposed CureLLM framework.
