Search papers, labs, and topics across Lattice.
This paper investigates the variability in answers provided by a retrieval-augmented generation (RAG) system when querying a multi-author institutional corpus, highlighting the critical issue of source-dependence in NLP evaluations. By shifting the focus from mere answer correctness to the relationships between sources, the authors develop a comprehensive framework that includes a benchmark (TransplantQA), a hierarchical retrieval strategy (HERO-QA), and a scoring system for inter-source relationships. The findings reveal that the extent of disagreement among sources is significantly underestimated, emphasizing the need for more nuanced evaluations in multi-source NLP applications across various domains.
Source-dependence in medical RAG systems reveals that answers can vary dramatically based on the source, challenging the single-gold-answer paradigm.
A retrieval-augmented generation (RAG) system deployed over a multi-author institutional corpus can give a different answer to the same question depending on which source it retrieves -- a failure mode the dominant single-gold-answer paradigm cannot diagnose. We argue that source-dependence is a missing axis of NLP evaluation, and that auditing it means shifting the unit of evaluation from answer correctness to the inter-source relationship. We make this concrete in transplant patient education, where institutional sources demonstrably disagree, releasing three artefacts: TransplantQA, a benchmark of real patient questions, each answered by grounding generation in multiple institutional handbooks as candidate sources; HERO-QA, a hierarchical retrieval strategy that grounds and audits each answer; and a structured-output judge that scores inter-source relationships on a validated 5-label taxonomy. At scale, better retrieval reveals far more disagreement than prior estimates suggested -- understating its prevalence, not its intensity. The framework is domain-agnostic and transfers to legal and educational RAG: measuring source-dependence is a responsibility for deployed multi-source NLP generally.
