Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces bangla-smollm-135m, a compact 135-million parameter foundational model specifically designed for efficient language modeling in the Bangla script, addressing the challenges posed by larger models in low-resource settings. By employing a novel deterministic intersect-and-append token merging strategy, the model effectively mitigates subword script fragmentation while maintaining the stability of pretrained parameters. In zero-shot evaluations across multiple benchmarks, bangla-smollm-135m not only matches but often outperforms larger models, demonstrating its competitive efficacy in the Bangla NLP landscape.
A 135M parameter model for Bangla outperforms larger counterparts, proving efficiency can rival scale in low-resource NLP tasks.
While the NLP landscape is dominated by multi-billion parameter architectures, their deployment in low-resource, non-Latin scripts remains computationally prohibitive for edge configurations, mobile systems, and decentralized local hardware. This paper presents bangla-smollm-135m, a highly compact 135-million parameter decoder-only foundational model engineered explicitly for high-efficiency language modeling in the Bangla script. By leveraging a deterministic intersect-and-append token merging strategy between TituLLMs and SmolLM2-135M, the model overcomes subword script fragmentation without destabilizing early pretrained parameter states. In zero-shot multi-task benchmark evaluations (PIQA_bn, OpenBookQA_bn, CommonsenseQA_bn, and Bangla_MMLU), bangla-smollm-135m matches or outperforms models twice its size (Gemma-3-270m) and achieves parity with models in the 1B parameter tier. The model is available at rnnandi/bangla-smollm-135m