Search papers, labs, and topics across Lattice.
This paper critiques the evaluation methods used for retrieval-augmented generation (RAG) systems, highlighting that traditional volume-based accuracy rewards systems for incorrect guesses. The authors introduce a penalty-aware evaluation framework that incorporates asymmetric scoring and knowledge-gap canaries to better assess the reliability of RAG outputs. Their findings reveal that while accuracy rates are similar across systems, the propensity to generate ungrounded answers varies significantly, suggesting that current evaluation practices may misrepresent system performance.
RAG systems may appear accurate, but they can generate ungrounded answers at alarming rates鈥攗p to 98.1% in some cases鈥攔evealing a critical flaw in traditional evaluation methods.
Volume-based accuracy rewards retrieval-augmented generation (RAG) systems for guessing: a system that answers everything outscores one that declines when its knowledge base cannot support an answer. Building on the confidence-target analysis of Kalai et al. (2025), we present a penalty-aware evaluation framework for deployed RAG products, combining (i) asymmetric scoring (correct +1, wrong -4, abstain 0), (ii) knowledge-gap canaries, questions whose answers are verifiably absent from the knowledge base, so that any answer constitutes ungrounded generation from parametric memory, and (iii) a failure-attribution pipeline that separates retrieval, generation, and abstention-policy failures. Applying the framework to three commercial RAG systems and a no-retrieval baseline on SimpleQA-Verified (1,000 questions x 3 repeats, graded blind by a cross-family three-judge panel with 98.9% unanimity), we find that accuracy when answering is closely clustered across systems (97.0-98.0%), while canary violation rates differ roughly sixfold (16.7% vs. 98.1%). The systems are separated less by what they answer correctly than by whether they answer at all when they should not, and penalty-aware scoring reorders the volume-based ranking accordingly; the reordering is stable across penalty settings from k=1 to k=9. All code, configurations, transcripts, and judge votes are released for independent audit.
