Search papers, labs, and topics across Lattice.
The CHALIS dataset was created to tackle the challenges of language identification in difficult scenarios, specifically focusing on cousin languages and orthographic noise. By collecting sentences from mutually intelligible language pairs and introducing various forms of noise, the dataset reveals significant shortcomings in existing language identification systems, particularly for lower-resource languages and transliterated text. Evaluation results indicate that all tested systems struggle considerably, highlighting the need for improved methodologies in this area.
Language identification systems falter dramatically when faced with cousin languages and orthographic noise, revealing critical gaps in current approaches.
We present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic noise. Our dataset has two parts: First, we collected sentences shared across mutually intelligible language pairs (Czech/Slovak, Spanish/Catalan, Portuguese/Galician, Danish/Norwegian). The second part tests for orthography noise: we transliterate text across multiple scripts, remove diacritics, simulate homoglyph attacks, and use Internet slang. We evaluate four widely used language identification systems on CHALIS and demonstrate that all struggle substantially in these scenarios, especially on lower-resource languages within cousin pairs and on transliterated input. The resource is publicly available at https://huggingface.co/datasets/michal-tichy/CHALIS.
