Search papers, labs, and topics across Lattice.
This paper investigates how the construction of Not Enough Information (NEI) examples in fact verification benchmarks impacts model performance and generalization. They introduce NEI-CAP, a diagnostic protocol that categorizes NEI examples by construction family to audit shortcut cues and assess cross-construction transfer. Experiments on SciFact, FEVER, and HoVer reveal that NEI competence does not reliably transfer across different construction methods, and that models often rely on shortcut cues related to specific NEI constructions.
Fact verification benchmarks may be teaching models the wrong thing: NEI labels are often gamed via dataset-specific shortcuts, meaning models don't actually learn to identify insufficient evidence.
Evidence absence is not evidence insufficiency, but fact verification benchmarks can make them observationally similar. The Not Enough Information (NEI) label is often operationalized through different evidence conditions, and that choice silently determines what a verifier learns and what its score can hide. We introduce NEI-CAP, a construction-aware diagnostic protocol for insufficient-evidence evaluation. Each NEI example carries the construction family that produced it; NEI-CAP audits shortcut cues, validates hard cases through human adjudication, and tests whether competence transfers across constructions. We instantiate the protocol in SciFact-style scientific verification, with FEVER and HoVer as bounded external controls. Across these settings, NEI competence does not transfer reliably: models trained on shortcut-prone constructions fail to recognize semantically related insufficient evidence, and mixed-construction training narrows but does not close the gap. Fixed-claim diagnostics further show that the evidence condition shifts confidence in the reference Support/Refute label, not only NEI recall, so an aggregate NEI score can hide which problem a model has actually solved.
