Search papers, labs, and topics across Lattice.
This study empirically evaluates the relevance of various Retrieval-Augmented Generation (RAG) metrics using a question-answering dataset derived from human-annotated business data. The authors compare scores from four distinct evaluation libraries against human evaluators and standard metrics like recall, revealing significant correlations and limitations in existing methodologies. The findings underscore the need for refined evaluation approaches in RAG systems, paving the way for future research in this area.
RAG metrics may not align with human judgment, revealing critical gaps in current evaluation practices.
This paper reports an empirical study evaluating the relevance of several RAG metrics. The experiment is based on a question-answering dataset created by human annotators from business data. The generated responses and retrieved spans of a RAG system are scored using evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik). These metrics are compared to scores given by two evaluators, as well as to standard metrics such as recall. An analysis of correlations is conducted. Finally, we highlight certain limitations of our methodology, compare it to those used in the literature, and suggest some avenues for future research. This paper is an English translation of a paper originally published in the French-speaking workshop EvalLLM (Brabant, 2026).
