Search papers, labs, and topics across Lattice.
This paper introduces GRAPHEVAL, a graph-based framework designed to quantify uncertainty and assess the coherence and robustness of reasoning in large language models (LLMs). By proposing the Graph Reasoning Coherence Score (GRCS), the authors reveal that this metric is consistently negatively correlated with reasoning faithfulness, highlighting the limitations of traditional evaluation methods like Self-Consistency. Additionally, the new Graph Self-Consistency (GSC) decoding strategy demonstrates improved reasoning fidelity while maintaining or enhancing accuracy in more capable models, thus addressing the shortcomings of naive majority voting in LLMs.
GRCS reveals that traditional evaluation methods can inflate perceived reasoning accuracy, exposing a critical gap in how we assess LLMs' logical validity.
Large-Language Models (LLMs) can be prone to flawed and unfaithful reasoning that decoding strategies like Self-Consistency (SC) fail to detect as they evaluate only final-answer agreement while ignoring the logical validity of intermediate steps. This raises three fundamental questions: How can we reliably quantify uncertainty in LLM reasoning? Can semantic, structural, and causal awareness select more faithful reasoning compared to na\"ive majority voting? and How robust is reasoning topology under adversarial conditions? To address these questions, we introduce GRAPHEVAL, a graph-based reasoning framework that re-frames uncertainty quantification (UQ) as a holistic reasoning fidelity problem. We propose a novel UQ metric, Graph Reasoning Coherence Score (GRCS), that quantifies semantic-structural consensus of the reasoning space and captures pathological mode collapse and confident hallucinations. We find that GRCS is the only metric that is consistently negatively correlated with reasoning faithfulness across both more capable and smaller models. Additionally, we introduce Graph Self-Consistency (GSC), a medoid-based decoding strategy that trades nominal accuracy for reasoning fidelity, exposing the degree to which SC is inflated by unfaithful lucky guesses in smaller models, while preserving or improving accuracy in more capable ones. Finally, through adversarial medoid ablation, we demonstrate that the GSC-selected path acts as a"load-bearing path"and forcing models away from it degrades reasoning faithfulness and, in targeted cases, causes drops in accuracy.
