Search papers, labs, and topics across Lattice.
This paper introduces BamiBERT, a novel BERT-based language model specifically designed for Vietnamese, which overcomes the limitations of the existing PhoBERT model. Trained on a substantial 129GB corpus for 20 epochs, BamiBERT features an extended context length of 2048 tokens and processes raw input without requiring external word segmentation. The model sets a new state of the art by outperforming PhoBERT on 11 out of 15 evaluation metrics across 8 Vietnamese benchmarks, showcasing its robust cross-domain generalization capabilities.
BamiBERT outperforms existing Vietnamese language models, achieving state-of-the-art results while simplifying input processing.
In this paper, we introduce BamiBERT, a new BERT-based pre-trained language model for Vietnamese that addresses key limitations of PhoBERT -- the current de facto Vietnamese text encoder. Trained from scratch on a 129GB corpus of general-domain Vietnamese text for 20 epochs, BamiBERT supports an extended context length of up to 2048 tokens and operates directly on raw input, eliminating the need for external word segmentation. Across 8 Vietnamese benchmarks, it achieves the best score on 11 of 15 metrics and the second-best on 3 others, setting a new state of the art among"base"-sized Vietnamese encoders and demonstrating strong cross-domain generalization. We release BamiBERT at: https://huggingface.co/Qualcomm-AI-Research/BamiBERT
