Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors developed SWORD, a benchmark across eight languages that generates syntactically valid but factually incorrect statements via controlled Wikidata triple perturbations to test whether LLMs can robustly detect and reject factual errors. Standard multilingual evaluations mask factual reasoning failures because they test for positive answer selection rather than true verification. The evaluation reveals that models paradoxically reject semantically plausible lies better than random nonsense, while suffering performance drops of up to 28 percentage points on East Asian languages despite achieving cross-lingual parity on standard benchmarks.
LLMs are counterintuitively worse at spotting nonsensical errors than plausible lies, and their ability to reject factual distortions collapses by up to 49% in East Asian languages despite showing parity on standard benchmarks.
Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating genuine factual understanding. We introduce Systematic Wikidata-based Object-Relation Distortion (SWORD), a benchmark that evaluates whether models consistently reject factual errors across languages. SWORD generates syntactically well-formed but factually incorrect statements in eight widely spoken languages through controlled perturbations of Wikidata triples, ranging from random entity substitutions to semantically plausible property-based selections. Our distortion-based evaluation surfaces two critical insights that remain entirely obscured by conventional benchmarks. First, models counterintuitively achieve higher accuracy on semantically plausible distortions than on nonsensical random substitutions, suggesting reliance on distributional familiarity rather than genuine factual verification. Second, models exhibiting comparable baseline accuracy across languages show substantial performance degradation specifically on (East) Asian languages when presented with distorted statements, with cross-lingual performance gaps reaching up to 28 percentage points (49\% relative reduction) in some models. These findings demonstrate that multilingual factual reasoning involves asymmetric capabilities that aggregate accuracy metrics systematically obscure.