Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces RARE, a framework for evaluating retrieval in RAG systems operating on corpora with high inter-document similarity and redundancy, like those found in finance, legal, and patent domains. RARE decomposes documents into atomic facts for precise redundancy tracking and uses a novel LLM-based data generation approach called CRRF (Criteria Rank-based Result Fusion) to improve the reliability of benchmark data. Experiments using RARE on Finance, Legal, and Patent corpora (resulting in the RedQA benchmark) reveal that retriever performance significantly degrades compared to standard benchmarks, highlighting robustness gaps in current evaluation methods.
Retriever performance can plummet from 66% to as low as 5% when evaluated on real-world, redundant corpora, exposing a critical flaw in current RAG benchmarks.
Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity. This mismatch undermines evaluation validity: retrievers can be unfairly undervalued even when they retrieve documents that provide sufficient evidence, because redundancy across documents is not accounted for in evaluation. On the other hand, retrievers that perform well on standard benchmarks often generalize poorly to real-world corpora with highly similar and redundant documents. We present RARE (Redundancy-Aware Retrieval Evaluation), a framework for constructing realistic benchmarks by (i) decomposing documents into atomic facts to enable precise redundancy tracking and (ii) enhancing LLM-based data generation with CRRF. RAG benchmark data usually requires multiple quality criteria, but LLMs often yield trivial outputs. CRRF scores criteria separately and fuses decisions by rank, improving the reliability of generated data. Applying RARE to Finance, Legal, and Patent corpora, we introduce RedQA, where a strong retriever baseline drops from 66.4% PerfRecall@10 on 4-hop General-Wiki to 5.0-27.9% PerfRecall@10 at 4-hop depth, revealing robustness gaps that current benchmarks fail to capture. RARE enables practitioners to build domain-specific RAG evaluations that faithfully reflect real-world deployment conditions.