Search papers, labs, and topics across Lattice.
This study conducts a comprehensive cross-benchmark evaluation of nine automated fact-checking (AFC) models, assessing their performance across diverse datasets in scientific, open-web, and climate domains. The findings reveal that fine-tuned models significantly outperform zero-shot LLMs and top AVeriTeC 2025 systems in certain contexts, while also highlighting the critical impact of domain and metric on model rankings. Additionally, the research confirms that the retrieval stage is a primary bottleneck, as replacing retrieved evidence with gold annotations leads to substantial improvements in veracity accuracy.
Fine-tuned models can outperform leading systems in automated fact-checking, but their effectiveness is highly dependent on the domain and evaluation metrics used.
Automated fact-checking (AFC) systems retrieve evidence and predict claim veracity, yet evaluations omit simple baselines, systems are developed for a single benchmark and cannot be trusted to generalise across domains. No prior work cross-evaluates the full two-stage retrieve-then-verify pipeline across diverse datasets, complementing retrieval-only studies (Thakur et al., 2021) and single-stage benchmarking studies (Calamai et al., 2025). We benchmark nine models, ranging from random and sparse baselines to fine-tuned transformers, zero-shot LLMs, and the two highest-ranked systems from the AVeriTeC 2025 shared task, across four datasets spanning scientific, open-web, and climate domains. Three findings stand out: (1) on ClimateCheck claim-only and fine-tuned models outperform zero-shot LLM and top-performing AVeriTeC 2025 systems, highlighting that noisy evidence can degrade veracity prediction; (2) system rankings are strongly domain- and metric-dependent: the best model on SciFact (macro-F1 0.70) drops to 0.31 on ClimateCheck, while the AVeriTeC 2025 winner and runner-up swap rankings based on evaluation metrics and datasets; (3) replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points across models, confirming retrieval remains primary bottleneck. We release code, pre-processed datasets, and all results to support reproducible AFC research.
