Search papers, labs, and topics across Lattice.
The paper introduces OpenLID-v3, an improved language identification system built upon OpenLID, designed to enhance precision in distinguishing closely related languages and filtering noise in multilingual datasets. They augmented the training data, merged problematic language variant clusters, and introduced a noise label. Evaluation against GlotLID on new benchmarks, particularly focusing on Bosnian/Croatian/Serbian, Romance varieties, and Scandinavian languages, demonstrates improved performance, although ensemble methods reduce coverage for low-resource languages.
OpenLID-v3 substantially boosts language identification precision for closely related languages, but at the cost of reduced coverage for low-resource languages when using ensemble methods.
Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks. During development, we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also substantially reduce coverage for low-resource languages. OpenLID-v3 is available on https://huggingface.co/HPLT/OpenLID-v3.
