Search papers, labs, and topics across Lattice.
This paper introduces ROMEVA, a method for enhancing multilingual language models like mBERT to better handle the morphological inconsistencies of Roman Urdu. By employing a combination of sub-word-average initialization and a PCA-guided anchor loss, ROMEVA stabilizes embeddings during vocabulary expansion, allowing for the integration of 500 fragmented tokens into the model. The results indicate that while ROMEVA effectively maintains the integrity of the pretrained embedding space, naive fine-tuning yields superior performance in sentiment classification, highlighting a trade-off between embedding stability and downstream task efficacy in low-resource languages.
ROMEVA reveals that prioritizing embedding stability can compromise downstream performance, challenging conventional wisdom in low-resource NLP.
Multilingual Language Models like mBERT are widely used for low-resource NLP, yet their adaptation to morphologically inconsistent languages such as Roman Urdu remains underexplored. Roman Urdu spelling variation causes severe sub-word fragmentation, averaging 1.50 sub-words per token. We propose \textit{ROMEVA} (Roman Urdu Embedding-preserving Vocabulary Adaptation), which combines sub-word-average initialization and a PCA-guided anchor loss to stabilize embeddings during vocabulary expansion. Using a 36,130-comment Roman Urdu corpus, we add 500 highly fragmented tokens to mBERT and compare naive fine-tuning, sub-word-aware fine-tuning, and \textit{ROMEVA}. While \textit{ROMEVA} most effectively preserves the pretrained embedding space, naive fine-tuning achieves the strongest downstream sentiment classification performance. These findings reveal a disconnect between embedding stability and downstream performance, suggesting that stronger adaptation may be preferable to strict embedding preservation in morphologically inconsistent languages.
