Search papers, labs, and topics across Lattice.
This paper introduces Chunk Coverage (CC), an oracle-independent criterion designed to evaluate the retrieval component of Retrieval-Augmented Generation (RAG) systems by measuring the fraction of corpus chunks retrieved across a test suite. The significance of CC lies in its ability to enhance test selection and generation, leading to more comprehensive coverage of the retrieval space and improved fault detection. In evaluations on clinical and financial RAG scenarios, CC-guided testing achieved 50% of attainable coverage 1.7 times faster than random selection and improved fault detection effectiveness by 10% to 25% compared to traditional methods.
CC-guided testing accelerates coverage and fault detection in RAG systems, revealing previously unexercised retrieval regions.
Retrieval-Augmented Generation (RAG)-based systems\footnote{For brevity, RAG-based systems are referred to as RAG systems throughout this paper.} are increasingly deployed in high-stakes settings where correct behaviour depends not only on the language model but also on the retrieval component that selects external documents at inference time. While existing RAG evaluation metrics assess retrieval and generation quality on a per-query basis, typically relying on query-level test oracles such as reference answers or relevance annotations, they provide limited insight into whether a test suite adequately exercises the retrieval behaviour of the system as a whole. In this paper, we introduce Chunk Coverage (CC), an oracle-independent test adequacy criterion for testing the retrieval component of RAG systems. CC measures the fraction of corpus chunks that are retrieved at least once across a test suite, providing a structural view of which parts of the retrieval space have been exercised. We further show how CC can be used to guide test selection and generation by prioritising queries that expand coverage of previously unexercised retrieval regions. We evaluate CC on clinical and financial RAG system scenarios. CC-guided testing reaches 50% of attainable coverage 1.7x faster than random selection and 4.2x faster than redundancy-biased strategies. Moreover, CC improves fault detection effectiveness (APFD) by 10% to 25% over random, indicating earlier discovery of distinct retrieval faults. These results show that CC captures retrieval diversity relevant to effective testing without requiring test oracles.
