Search papers, labs, and topics across Lattice.
The paper introduces RAG-X, a diagnostic framework for evaluating retrieval-augmented generation (RAG) systems in the context of medical question answering across information extraction, short-answer generation, and multiple-choice question answering tasks. RAG-X uses Context Utilization Efficiency (CUE) metrics to independently evaluate the retriever and generator, distinguishing between verified grounding and deceptive accuracy. Experiments using RAG-X reveal an "Accuracy Fallacy," demonstrating a significant discrepancy (14%) between perceived system accuracy and actual grounding in evidence.
RAG systems can appear accurate in medical QA while being deceptively ungrounded, with a 14% gap between perceived success and evidence-based grounding, a problem RAG-X helps diagnose.
Automated question-answering (QA) systems increasingly rely on retrieval-augmented generation (RAG) to ground large language models (LLMs) in authoritative medical knowledge, ensuring clinical accuracy and patient safety in Artificial Intelligence (AI) applications for healthcare. Despite progress in RAG evaluation, current benchmarks focus only on simple multiple-choice QA tasks and employ metrics that poorly capture the semantic precision required for complex QA tasks. These approaches fail to diagnose whether an error stems from faulty retrieval or flawed generation, limiting developers from performing targeted improvement. To address this gap, we propose RAG-X, a diagnostic framework that evaluates the retriever and generator independently across a triad of QA tasks: information extraction, short-answer generation, and multiple-choice question (MCQ) answering. RAG-X introduces Context Utilization Efficiency (CUE) metrics to disaggregate system success into interpretable quadrants, isolating verified grounding from deceptive accuracy. Our experiments reveal an ``Accuracy Fallacy", where a 14\% gap separates perceived system success from evidence-based grounding. By surfacing hidden failure modes, RAG-X offers the diagnostic transparency needed for safe and verifiable clinical RAG systems.
