Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces NewsScope, a new dataset and benchmark for schema-grounded news claim extraction across diverse domains, comprising 455 articles with in-domain and out-of-source splits. They fine-tuned LLaMA 3.1 8B using LoRA on a subset of the dataset, achieving 89.4% human-evaluated accuracy, comparable to GPT-4o-mini, and outperforming it on political claims. A numeric grounding filter further enhances accuracy, and the open-weight model allows for cost-effective deployment.
An open-source LLaMA 3.1 8B model rivals GPT-4o-mini in schema-grounded news claim extraction, even surpassing it in the politically charged domain.
Automated news verification requires structured claim extraction, but existing approaches either lack schema compliance or generalize poorly across domains. This paper presents NewsScope, a cross-domain dataset, benchmark, and fine-tuned model for schema-grounded news claim extraction. The dataset contains 455 articles across politics, health, science/environment, and business, consisting of 395 in-domain articles and 60 out-of-source articles for generalization testing. LLaMA 3.1 8B was fine-tuned using LoRA on 315 training examples and evaluated on held-out in-domain (80 articles) and out-of-source (60 articles) test sets. Human evaluation on 400 claims shows NewsScope achieves 89.4% human-evaluated accuracy compared to GPT-4o-mini's 93.7% (p=0.07). NewsScope outperforms GPT-4o-mini on political claims (94.3% vs. 87.8%). A numeric grounding filter further improves accuracy to 91.6%, narrowing the gap to 2.1 percentage points. Inter-annotator agreement studies (160 claims) confirm labeling reliability (94.6% positive agreement on SUPPORTED judgments). The open-weight model enables offline deployment at approximately $15 on-demand compute (or $0 on free tiers). Code and benchmark are publicly released.