Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces an incremental pooled LLM evaluation method for selecting retrieval models in production RAG systems, addressing the challenges of obtaining scalable and repeatable relevance judgments. By leveraging LLMs to assess the union of documents from various candidate systems and incrementally expanding the pool with new contributions, the approach significantly reduces evaluation costs while maintaining strong correlations with gold-standard benchmarks. The method demonstrates a remarkable 65-80% reuse of judgments and a potential cost reduction of up to 4.9 times, making it a viable solution for continuous model evaluation in dynamic environments.
Achieving up to 80% judgment reuse, this method slashes evaluation costs and streamlines retrieval model selection in real-world applications.
Selecting a retrieval model for a production RAG system requires reliable comparative evaluation, but obtaining relevance judgments at scale is expensive and difficult to repeat as new candidate systems arrive. We study pooled LLM evaluation, in which an LLM judges the union of documents retrieved by the current set of candidate systems, and the pool is then expanded incrementally as new systems are introduced by judging only the new documents they contribute. These judgments are reused to evaluate all systems on a common basis. We validate this approach on four retrieval benchmarks with 11 systems spanning dense, sparse, and hybrid configurations, and deploy it to compare 62 retrieval configurations for a financial news QA system. Pooled LLM rankings correlate strongly with gold-standard evaluation across datasets, and 97% of pairwise system orderings are preserved once bootstrap uncertainty in the qrels is taken into account. In production, document overlap yields 65-80% judgment reuse and up to 4.9x lower evaluation cost, allowing teams to benchmark new retrieval candidates without re-judging previously assessed documents. These results suggest pooled LLM evaluation is a practical and cost-effective workflow for incremental retrieval model selection in deployed systems.