Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Eval-Pair Matrix, a novel meta-evaluation protocol designed to address self-leniency issues in LLM-as-a-judge evaluations for retrieval-augmented generation (RAG). By inducing hidden answer-causal contradictions and utilizing the same models for both answer generation and evaluation, the authors reveal that traditional comparisons of diagonal and off-diagonal F1 scores may obscure critical insights about model performance. The findings indicate that while same-model recall effects are negligible, there is a significant drop in matching-judge flagging for answers that successfully avoid the induced claims, emphasizing the need for more nuanced evaluation metrics in RAG studies.
Traditional evaluation methods may mislead researchers about LLM performance, as this study reveals minimal recall differences when using the same model for generating and judging answers.
LLM-as-a-judge evaluation is widely used for retrieval-augmented generation (RAG), but reusing the same model family as both generator and judge makes self-leniency difficult to identify. We introduce Eval-Pair Matrix, a controlled meta evaluation protocol for source-grounded RAG. Starting from GaRAGe questions and grounding passages, we induce one hidden answer-causal contradiction per record, generate answers from perturbed passages with GPT, Grok, and Gemini models, and then use the same models as blind judges to evaluate each answer against the original passages. The experiment contains 300 core records, 897 labeled generator outputs, and 2,683 judge verdicts in a crossed 3 x 3 matrix; the primary analysis uses 275 fully validated records. Instead of comparing diagonal and off-diagonal cells across different answers, we estimate same-model effects by pairing judges on the exact same candidate answer. This changes the interpretation: diagonal and off diagonal F1 are similar, and the paired same-model recall effect is near zero (-0.5 pp; 95% cluster bootstrap CI [-2.7, +1.7]). The only robust paired gap is lower matching-judge flagging for answers that avoided the induced claim (-4.3 pp). A targeted human evaluation finds that reviewed apparent false positives are alternate source-error detections, mistakes in labeling whether the induced claim was adopted, or unclear cases; none were adjudicated as genuine false alarms. The lesson is methodological: RAG judge studies should report full matrices, answer-paired effects, behavior strata, and label-task alignment.