Search papers, labs, and topics across Lattice.
This study introduces SemCog Bench, a benchmark designed to evaluate large language models (LLMs) on their ability to identify cognates and disambiguate meanings between Arabic and Hebrew, two closely related Semitic languages. The evaluation reveals that while LLMs perform well on true cognates, they struggle significantly with false friends and loanwords, indicating a reliance on surface-form similarity rather than deeper semantic understanding. The findings highlight a critical gap in cross-lingual reasoning capabilities of LLMs and underscore the limitations of contextual cues in resolving form-meaning conflicts.
LLMs excel at recognizing true cognates but falter dramatically with false friends, exposing a fundamental flaw in cross-lingual semantic reasoning.
Arabic and Hebrew, as closely related Semitic languages, share a substantial lexicon of true cognates, misleading false friends, and modern loanwords. This overlap poses a challenge for cross-lingual semantic understanding in large language models (LLMs). To evaluate this capability, we introduce SemCog Bench, a curated benchmark of 1,858 Arabic--Hebrew word pairs with sentence-level annotations for cognate identification and semantic disambiguation. We evaluate open-source and commercial LLMs across multiple input representations (raw, diacritized, Romanized, and phonetic) and reveal a critical gap in cross-lingual reasoning. While models achieve high accuracy on true cognates, performance drops sharply on false friends and loanwords, reflecting a strong reliance on surface-form similarity. Furthermore, sentence-level context yields only modest improvements, suggesting that contextual cues alone are insufficient to overcome misleading form-based signals. These findings reveal a fundamental limitation of current LLMs in resolving cross-lingual form--meaning conflicts and establish SemCog Bench as a rigorous benchmark for multilingual semantic reasoning. Our code and data are publicly available.
