Search papers, labs, and topics across Lattice.
The authors introduce KS-PRET-5M, a new 5 million word, 12 million token pretraining dataset for the Kashmiri language. The dataset was created from digitized archival material and web sources, processed through an extensive cleaning pipeline to minimize script contamination. Tokenization using google/muril-base-cased resulted in a subword ratio of 2.383 tokens per word, significantly higher than estimates from related languages.
Kashmiri NLP gets a major boost: a new 5M-word pretraining dataset dwarfs existing resources and promises to unlock better language models.
We present KS-PRET-5M, the largest publicly available pretraining dataset for the Kashmiri language, comprising 5,090,244 (5.09M) words, 27,692,959 (27.6M) characters, and a vocabulary of 295,433 (295.4K) unique word types. We assembled the dataset from two source classes: digitized archival and literary material, encompassing literature, news, biographies, novels, poetry, religious scholarship, and academic writing, recovered from the proprietary InPage desktop-publishing format using the converter of Malik~\cite{malik2024inpage}, and Unicode-native text collected from Kashmiri-language web sources. All text was processed through an eleven-stage cleaning pipeline that achieves a mean Kashmiri script ratio of 0.9965, reducing Devanagari contamination to 146 characters across the full dataset. We tokenized the dataset empirically using google/muril-base-cased, yielding a subword ratio of 2.383 tokens per word and a total of approximately 12.13 million subword tokens, substantially higher than prior estimates derived from non-Kashmiri Perso-Arabic analogues. KS-PRET-5M is released as a single continuous text stream under CC~BY~4.0 to support language model pretraining, tokenizer training, and computational linguistic research for Kashmiri.
