Search papers, labs, and topics across Lattice.
This paper introduces MEDIAREF, a publicly available knowledge store designed to facilitate media background checks (MBCs) for assessing the credibility of evidence sources in retrieval-augmented generation (RAG) systems used for automated fact-checking. By providing a low-cost and reproducible method for evaluating MBC generation across 200 media sources, MEDIAREF addresses the limitations of existing approaches that rely on proprietary search APIs. The results show that LLMs utilizing MEDIAREF achieve higher-quality MBC generation, enhancing the transparency and reliability of automated fact-checking systems.
MEDIAREF transforms the landscape of automated fact-checking by providing a free, reproducible resource that boosts the credibility of evidence sourcing in LLMs.
LLM-based retrieval-augmented generation (RAG) is increasingly used for automated fact-checking (AFC) and related tasks. By grounding LLM outputs in retrieved evidence, RAG-based systems provide transparent justifications while allowing external information to be updated independently of the underlying model. However, existing approaches often assume retrieved evidence is reliable, although real-world information may be conflicting, outdated, and can originate from unreliable or biased sources. Recent work on *source-critical reasoning* addresses this challenge through media background checks (MBCs) (Schlichtkrull, 2024), which assess the credibility of evidence sources to support downstream fact verification. However, generating MBCs relies on costly proprietary search APIs, limiting reproducibility. To mitigate this issue, we introduce MEDIAREF, a publicly available knowledge store of web-sourced documents that enables reproducible, low-cost evaluation of MBC generation across 200 media sources. We describe a reproducible methodology for constructing and updating the collection, assess widely used LLMs on the MBC generation task, and demonstrate that MEDIAREF supports higher-quality MBC generation through both automatic and qualitative evaluation.
