Search papers, labs, and topics across Lattice.
To resolve the blind spot where standard overlap and perplexity metrics fail to distinguish memorized associations from grounded reasoning, the authors establish a knowledge graph-based evaluation pipeline for question answering. The framework introduces Semantic Structural Similarity for KGs (S3KG), a continuous scoring function that blends topological graph structure with semantic node representations alongside a diagnostic reasoning error taxonomy. Evaluated against traditional metrics on a curated QA benchmark, S3KG yields substantially more interpretable, granular assessments of model correctness and contextual faithfulness.
By mapping context and generation into knowledge graphs, S3KG untangles whether an LLM is genuinely reasoning over provided context or merely parroting memorized pretraining associations.
Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.
