Search papers, labs, and topics across Lattice.
This paper introduces AtomEval, a validity-aware evaluation framework for fact verification that decomposes claims into subject-relation-object-modifier (SROM) atoms. AtomEval uses Atomic Validity Scoring (AVS) to more accurately detect factual corruptions introduced by adversarial rewrites, addressing the limitations of standard metrics. Experiments on the FEVER dataset demonstrate that AtomEval provides more reliable evaluation signals, revealing that stronger LLMs don't always generate more effective adversarial claims when validity is considered.
Current adversarial evaluation practices are misleading: stronger LLMs don't necessarily produce more effective adversarial claims when you actually check if the facts are still true.
Adversarial claim rewriting is widely used to test fact-checking systems, but standard metrics fail to capture truth-conditional consistency and often label semantically corrupted rewrites as successful. We introduce AtomEval, a validity-aware evaluation framework that decomposes claims into subject-relation-object-modifier (SROM) atoms and scores adversarial rewrites with Atomic Validity Scoring (AVS), enabling detection of factual corruption beyond surface similarity. Experiments on the FEVER dataset across representative attack strategies and LLM generators show that AtomEval provides more reliable evaluation signals in our experiments. Using AtomEval, we further analyze LLM-based adversarial generators and observe that stronger models do not necessarily produce more effective adversarial claims under validity-aware evaluation, highlighting previously overlooked limitations in current adversarial evaluation practices.
