Search papers, labs, and topics across Lattice.
SiDiaC-v.2.0, the largest Sinhala Diachronic Corpus to date, was created by expanding and refining SiDiaC-v.1.0 with texts digitized using Google Document AI OCR and post-processed for formatting, code-mixing, and token errors. The corpus spans 1800-1955 CE publication dates and 5th-20th century CE written dates, comprising 244k words across 185 works, with a 70k-word subset annotated by written date. Informed by corpora like FarPaHC and CCOHA, SiDiaC-v.2.0 is categorized into primary (Fiction/Non-Fiction) and secondary (e.g., Religious, History) genres, providing a valuable resource for Sinhala NLP despite resource constraints.
A new, large-scale diachronic corpus for Sinhala, SiDiaC-v.2.0, offers a crucial resource for NLP research on this low-resource language, enabling studies of linguistic change and historical text analysis.
SiDiaC-v.2.0 is the largest comprehensive Sinhala Diachronic Corpus to date, covering a period from 1800 CE to 1955 CE in terms of publication dates, and a historical span from the 5th to the 20th century CE in terms of written dates. The corpus consists of 244k words across 185 literary works that underwent thorough filtering, preprocessing, and copyright compliance checks, followed by extensive post-processing. Additionally, a subset of 59 documents totalling 70k words was annotated based on their written dates. Texts from the National Library of Sri Lanka were selected from the SiDiaC-v.1.0 non-filtered list, which was digitised using Google Document AI OCR. This was followed by post-processing to correct formatting issues, address code-mixing, include special tokens, and fix malformed tokens. The construction of SiDiaC-v.2.0 was informed by practices from other corpora, such as FarPaHC, SiDiaC-v.1.0, and CCOHA. This was particularly relevant for syntactic annotation and text normalisation strategies, given the shared characteristics of low-resource language status between Faroese and the similar cleaning strategies utilised in CCOHA. This corpus is categorised into two layers based on genres: primary and secondary. The primary categorisation is binary, assigning each book to either Non-Fiction or Fiction. The secondary categorisation is more detailed, grouping texts under specific genres such as Religious, History, Poetry, Language, and Medical. Despite facing challenges due to limited resources, SiDiaC-v.2.0 serves as a comprehensive resource for Sinhala NLP, building upon the work previously done in SiDiaC-v.1.0.
