Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce CommunityFact, a dynamic benchmark for misinformation detection comprising 15,992 claims across five languages and two domains, designed to reflect real-world online settings. They evaluated ten LLMs with and without web access, finding that while web access improves performance, LLMs' source selection often diverges from human raters. Furthermore, they show that Community Notes can be used as a training signal to improve claim-conditioned source suggestion.
LLMs with web access still struggle to align their source selection for misinformation detection with human consensus, revealing a critical gap in real-world fact verification.
Misinformation verification increasingly occurs in public, fast-moving, and multilingual online settings, where static benchmarks provide an incomplete measure of model reliability. We introduce CommunityFact, a refreshable benchmark for misinformation detection in the wild, with three major goals: coverage, granularity, and redistributability. This release contains 15,992 standalone claims across five languages and two domains. We evaluate ten LLMs under varying inference-time capabilities, including thinking and web-search. Our results show that closed-input verification remains challenging, web access yields the largest gains, and web-enabled LLMs'source-selection policies are systematically misaligned with the sources human Community Notes raters converge on -- a gap that closes through model-specific mechanisms of retrieval expansion or pruning. We further find substantial variation across language-domain slices and across the evidence ecosystems used by web-enabled systems. Beyond evaluation, CommunityFact positions Community Notes as a training signal for claim-conditioned source suggesters that could improve factual verification on novel claims.