Search papers, labs, and topics across Lattice.
This paper introduces GraphInfer-Bench, a novel benchmark designed to evaluate the inference capabilities of large language models (LLMs) on graph-based tasks that require synthesizing information from multiple nodes rather than retrieving it from a single source. The benchmark includes five tasks focused on Description and Comparison, with a dataset of 42,000 samples derived from six real-world graphs, rigorously screened for quality. Evaluation results reveal that while graph-token alignment models and frontier LLMs show some strengths, plain graph neural networks (GNNs) consistently outperform LLMs, particularly in community detection, highlighting a significant gap in LLMs' graph inference capabilities.
No existing LLM architecture can fully bridge the gap in graph inference tasks, with plain GNNs outperforming even the most advanced LLMs in critical areas like community detection.
Graph analysis underlies many applications whose answers cannot be looked up in a single record or retrieved along a path: laundering rings, drug repurposing, user preference, and scientific theme are all inferred from a node together with its neighbourhood. We introduce GraphInfer-Bench, a benchmark for whether LLMs can perform this graph inference: producing an open-ended answer that no single node supports and no path retrieves. Existing graph-QA protocols cannot test this capability: algorithm simulation, node classification, single-node description, KG-QA, and GraphRAG all admit answers retrievable from one node or along a path. GraphInfer-Bench defines five tasks along Description (what a region is) and Comparison (how regions differ), each constructed so the ground truth lives in no single node. The release contains 42,000 samples across six real-world graphs, produced automatically and screened by a four-layer quality-control protocol. We evaluate four method families against the same tasks: graph-token alignment models, zero-shot frontier closed-source LLMs, Graph2Text supervised fine-tuning, and plain GNNs as a structural reference. No method family closes the gap. Graph-token alignment partially handles description tasks (relational, theme) but collapses on comparison tasks. Frontier LLMs lead on outlier detection and community partition among LLM-based methods but lag on masked-node prediction. Graph2Text SFT is the strongest LLM-based method on the description side yet falls behind frontier LLMs on comparison. Across every task, plain GNNs match or beat the strongest LLM-based row, with the largest margin on community detection. GraphInfer-Bench surfaces graph inference as an open capability gap rather than a property of any one architecture.
