Search papers, labs, and topics across Lattice.
This paper introduces a metamorphic testing framework designed to evaluate Retrieval-Augmented Generation (RAG) systems against evolving document corpora, addressing the shortcomings of static evaluation methods like RAGAS. By formalizing a fault taxonomy and implementing 11 mutation operators, the authors systematically perturb both the retrieval index and the retrieved context, revealing significant metamorphic violation rates between 4.9% and 10.2%. The proposed approach outperforms existing metrics, achieving F1 scores between 0.927 and 1.000, thereby providing critical insights for improving RAG system robustness amid data changes.
Metamorphic testing reveals that RAG systems can miss up to 10% of faults when document corpora evolve, exposing a critical gap in current evaluation practices.
Retrieval-Augmented Generation (RAG)-based LLM systems rely on external document corpora that can evolve and change over time. However, current evaluation methodologies (e.g., RAGAS) assess correctness against static snapshots, failing to detect faults when routine updates, factual changes, or noise alter the underlying data. We introduce a metamorphic testing framework that evaluates the consistency of RAG systems under corpus evolution. We formalise a fault taxonomy and 11 mutation operators that systematically perturb the system at both the pre-chunk (retrieval index) and post-chunk (retrieved context) levels. An empirical evaluation across five datasets and over 28k mutants reveals metamorphic violation rates of 4.9-10.2%. In a meta-evaluation against ground truth, our metamorphic oracle achieves F1 scores of 0.927-1.000, while the best RAGAS metric reaches only 0.570. Finally, we provide actionable insights into mitigating these faults through retrieval re-configuration, generator upgrades, and LLM-based reranking.
