Search papers, labs, and topics across Lattice.
This study addresses the challenge of word segmentation in the extinct Tangut language, which lacks explicit word boundaries, by integrating expert-annotated segments, traditional lexicons, and unlabeled text. The authors employ a reliability-calibrated lexicon-lattice representation alongside a lightweight character encoder pretrained with masked language modeling (MLM), achieving a CRF F1 score of approximately 0.91. Notably, the TangutEncoder not only reaches a mean F1 score of 0.911 but also enhances recall beyond the limited supervised vocabulary, indicating its potential for generalization across diverse text passages.
Achieving a mean F1 score of 0.911 in Tangut word segmentation reveals the power of combining traditional lexicons with modern machine learning techniques in resource-scarce scenarios.
Tangut is an extinct language whose script does not explicitly mark word boundaries. We present the first systematic study of Tangut word segmentation using 2,750 expert-annotated segments(31,893 tokens), traditional lexicons, and unlabeled text. Our framework combines a reliability-calibrated lexicon-lattice representation, explicit distributional statistics, and a lightweight character encoder pretrained with MLM. Segment-level five-fold cross-validation shows that lexical and statistical features raise CRF F1 to approximately 0.91. The full TangutEncoder reaches the highest mean F1 (0.911) and improves recall beyond the labeled training vocabulary. These results demonstrate generalization beyond the limited supervised vocabulary across thematically diverse held-out passages, while document-level transfer remains to be evaluated.
Tsinghua AI