Search papers, labs, and topics across Lattice.
This meta-analysis examines the impact of LLMs on IR benchmarks, specifically TREC Robust04 and TREC DL20, by analyzing 143 publications. Results show an "LLM effect," with LLM-based systems achieving significantly higher nDCG@10 scores compared to pre-LLM baselines. However, the study also detects measurable data contamination in both benchmarks, making it difficult to definitively attribute the performance gains to methodological advances rather than memorization.
LLMs are inflating IR benchmark scores, but it's hard to tell how much is real progress versus just memorization.
Benchmark collections have long enabled controlled comparison and cumulative progress in Information Retrieval (IR). However, prior meta-analyses have shown that reported effectiveness gains often fail to accumulate, in part due to the use of weak or outdated baselines. While large language models are increasingly used in retrieval pipelines, their impact on established IR benchmarks has not been systematically analyzed. In this study, we analyze 143 publications reporting results on the TREC Robust04 collection and the TREC Deep Learning 2020 (DL20) passage retrieval benchmark to examine longitudinal trends in retrieval effectiveness and baseline strength. We observe what we term an \emph{LLM effect}: recent systems incorporating LLM components achieve 8.8\% higher nDCG@10 on DL20 compared to the best result from TREC 2020 and approximately 20\% higher on Robust04 since 2023. However, adapting a data contamination detection approach to reranking reveals measurable contamination in both benchmarks. While excluding contaminated topics reduces effectiveness, confidence intervals remain wide, making it difficult to determine whether the LLM effect reflects genuine methodological advances or memorization from pretraining data.
