Search papers, labs, and topics across Lattice.
The paper analyzes 9 claim verification datasets using GPT-4o-mini to generate structured reasoning traces, revealing that most examples rely on direct evidence extraction rather than complex reasoning. A dataset-level analysis exposes significant biases, with some datasets primarily testing lexical matching and others requiring information synthesis. Using a 1B-parameter reasoning verifier, the authors identify domain-specific error profiles, suggesting that current benchmarks overemphasize retrieval-plus-entailment ability.
Claim verification benchmarks may be fooling us: they overwhelmingly test simple evidence retrieval, not the complex reasoning we thought they did.
Despite rapid progress in claim verification, we lack a systematic understanding of what reasoning these benchmarks actually exercise. We generate structured reasoning traces for 24K claim-verification examples across 9 datasets using GPT-4o-mini and find that direct evidence extraction dominates, while multi-sentence synthesis and numerical reasoning are severely under-represented. A dataset-level breakdown reveals stark biases: some datasets almost exclusively test lexical matching, while others require information synthesis in roughly half of cases. Using a compact 1B-parameter reasoning verifier, we further characterize five error types and show that error profiles vary dramatically by domain -- general-domain verification is dominated by lexical overlap bias, scientific verification by overcautiousness, and mathematical verification by arithmetic reasoning failures. Our findings suggest that high benchmark scores primarily reflect retrieval-plus-entailment ability. We outline recommendations for building more challenging evaluation suites that better test the reasoning capabilities verification systems need.
