Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of graph evidence in graph-augmented large language models (LLMs) by employing HopQA, a diagnostic task that requires identifying the shortest-hop distance between query nodes. The authors reveal that simply providing graph evidence does not guarantee its usability by the native decoder, as evidenced by the poor performance of existing models on this task. They introduce S$^2$GE, a novel approach that enhances decoder usability through query-aware sampling and structure-preserving alignment, achieving significant performance improvements across multiple datasets.
Graph-augmented LLMs fail to leverage provided graph evidence effectively, revealing a critical gap in their design that can be addressed with targeted interventions.
Graph-augmented large language models often assume that graph evidence produced by external computation and placed in the input can be used by the native decoder. We test this assumption with HopQA, a deliberately bounded diagnostic that asks for the shortest-hop distance between two query nodes. Because the answer is a small integer and the target is purely topological, failure cannot be dismissed as open-ended generation or ambiguous evaluation. Yet existing graph-augmented baselines still fail on this setting, showing that providing graph evidence is not the same as making it usable. We introduce an intervention triangle with three matched conditions: readable graph evidence, shuffled graph evidence, and no-graph input. This separates evidence inclusion, structural readability, and decoder-usable topology. Guided by this diagnosis, we present S$^2$GE as an instance showing that diagnosis-driven interface design can improve native decoder usability. S$^2$GE uses query-aware sampling, endpoint and proximity-based ordering, and structure-preserving alignment. Across DBLP, Biomedical, GoodReads, and PubMed, S$^2$GE achieves strict exact-match scores of $36.5\%$, $57.8\%$, $76.6\%$, and $52.0\%$, improving over the strongest native-generation baseline by $53.5$ points on average. The interventions further reveal harmful-shuffle, shuffle-robust, and no-graph-saturated regimes.
