Search papers, labs, and topics across Lattice.
This paper introduces HelaBERT, a family of BERT-based models specifically pre-trained on a substantial corpus of Sinhala text, including news articles and web data, to enhance language understanding in this underrepresented language. The models, HelaBERT-Small and HelaBERT-Large, incorporate a novel dual pooling classification head, which significantly improves performance on sentiment analysis and moderately enhances news category classification. Evaluations across four classification tasks demonstrate that while the dual pooling head excels in certain areas, the traditional [CLS]-linear head remains effective for tasks with shorter inputs, highlighting the nuanced performance characteristics of these models.
HelaBERT achieves notable improvements in Sinhala NLP tasks, particularly in sentiment analysis, by leveraging a dual pooling classification head.
We present HelaBERT, a family of two BERT-based masked language models pre-trained from scratch on approximately 1 billion tokens of Sinhala text sourced from MADLAD-400, CulturaX, and a custom corpus comprising news articles, Sinhala Wikipedia, and web crawl data. HelaBERT-Small (~23.3M parameters, 6 layers) and HelaBERT-Large (~110M parameters, 12 layers) both use a SentencePiece Unigram tokenizer (vocabulary size 32,000) tailored to Sinhala's agglutinative morphology and complex script. We evaluate both models on four downstream Sinhala text classification tasks: news category classification, news source classification, sentiment analysis, and writing style classification, using 5 independent seed runs with stratified 80/20 train/test splits. We additionally propose a dual pooling classification head and evaluate it systematically across all four tasks, finding consistent improvements on sentiment analysis and a moderate gain on news category classification for HelaBERT-Small, while the standard [CLS]-linear head remains competitive on news source classification, a headline-level task with short average input length. We release both models to support further research in Sinhala NLP.
