Search papers, labs, and topics across Lattice.
This paper introduces CKTN, the first comprehensive multilingual corpus for the Cham, Khmer, and Tay-Nung languages, addressing the unique challenges posed by their distinct scripts and contact with Vietnamese. The authors demonstrate that existing multilingual encoders inadequately handle these languages, often leading to misleading adaptation metrics that overlook semantic generalization. By employing a script-aware adaptation strategy that combines vocabulary augmentation with calibrated replaced-token pretraining, they achieve significantly improved classification performance and reduced fragmentation in language representation.
Existing multilingual encoders can mislead researchers by fragmenting minority languages, but a new corpus and method reveal their true potential.
Vietnam's ethnic minority languages are almost absent from the field of Natural Language Processing (NLP), and the challenge goes beyond data scarcity: Cham, Khmer, and Tay-Nung differ sharply in script, Vietnamese contact, and standardization, conditions under which standard multilingual adaptation can learn the wrong signals. We introduce CKTN, the first corpus and benchmark for these languages (44,367 documents, 24M subword tokens), spanning continued pretraining, category classification, and summary-document retrieval. We show that existing multilingual encoders severely fragment these languages, and that common adaptation metrics can mislead: models may lower language-modeling loss or excel at lexical-overlap retrieval while still failing at semantic generalization across documents. We address this with a script-aware adaptation recipe - vocabulary augmentation combined with calibrated replaced-token pretraining - that prevents the discriminator from exploiting trivial script mismatches. The result is an encoder with substantially less fragmentation and the strongest classification performance among evaluated models, exposing the limits of lexical-overlap retrieval as an evaluation signal.
