Search papers, labs, and topics across Lattice.
The authors introduce MultiCW, a large-scale, balanced, multilingual dataset for check-worthy claim detection, comprising 123,722 samples across 16 languages, 7 domains, and 2 writing styles, along with a 27,761 sample out-of-distribution evaluation set. They benchmark 3 fine-tuned multilingual transformers and 15 LLMs (zero-shot) on MultiCW, demonstrating the superior performance of fine-tuned models and their strong out-of-distribution generalization capabilities. The dataset and benchmarking aim to facilitate advancements in automated fact-checking and systematic evaluation of claim detection models.
Forget prompting: fine-tuning multilingual transformers on the new MultiCW dataset beats zero-shot LLMs at detecting check-worthy claims, even across languages and domains.
Large Language Models (LLMs) are beginning to reshape how media professionals verify information, yet automated support for detecting check-worthy claims a key step in the fact-checking process remains limited. We introduce the Multi-Check-Worthy (MultiCW) dataset, a balanced multilingual benchmark for check-worthy claim detection spanning 16 languages, 7 topical domains, and 2 writing styles. It consists of 123,722 samples, evenly distributed between noisy (informal) and structured (formal) texts, with balanced representation of check-worthy and non-check-worthy classes across all languages. To probe robustness, we also introduce an equally balanced out-of-distribution evaluation set of 27,761 samples in 4 additional languages. To provide baselines, we benchmark 3 common fine-tuned multilingual transformers against a diverse set of 15 commercial and open LLMs under zero-shot settings. Our findings show that fine-tuned models consistently outperform zero-shot LLMs on claim classification and show strong out-of-distribution generalization across languages, domains, and styles. MultiCW provides a rigorous multilingual resource for advancing automated fact-checking and enables systematic comparisons between fine-tuned models and cutting-edge LLMs on the check-worthy claim detection task.
