Search papers, labs, and topics across Lattice.
This study investigates the impact of knowledge-graph (KG) grounding on large language models (LLMs) in the context of clinical question answering, revealing that KG grounding only enhances performance when dealing with out-of-training knowledge. The authors reproduce and clarify the original HealthBench scores, highlighting a significant discrepancy in grading that affects the perceived efficacy of LLMs versus retrieval-augmented tools. Their findings indicate that while KG grounding does not improve performance on known facts, it dramatically boosts accuracy on novel information, underscoring the importance of training data relevance in LLM applications.
Grounding LLMs in knowledge graphs only pays off when the information is outside their training set, revealing a critical limitation in current clinical AI tools.
A recent Nature Medicine study reports that general-purpose frontier LLMs outperform specialized retrieval-augmented clinical tools on medical benchmarks, and that retrieval can hurt strong models. We ask the natural follow-up: does structured knowledge-graph (KG) grounding change this, and when does grounding help at all? We contribute two results. First, a reproduction: the study's headline HealthBench score (~88) is the Consensus variant, not full HealthBench, where frontier models and ideal completions both score ~46-47 under a physician-calibrated grader (agreement 82.5%); we reproduce GPT-5.2 Consensus =90.9 and flag a score-deflating grader bug. Second, a knowledge-boundary result. Using a graph+vector engine (samyama-graph) over the public biomedical KG PrimeKG, neither naive triple retrieval nor an agentic natural-language-to-Cypher loop (82% successful queries) improves MedQA across a weak-to-strong model ladder (all |Delta| <= 3.4). On a synthetic counterfactual KG, and on a hybrid benchmark mixing known and novel facts, the identical pipeline lifts out-of-training accuracy from chance to ~100% (+68 to +79) while adding nothing on known facts (a no-LLM arm answers both). Across three regimes (no-knowledge, graph-aided, hybrid), grounding helps only insofar as the decisive fact lies outside the model's training -- public-KG facts are redundant, private and novel data are where it pays -- matching the study's institutional-data caveat.
