Search papers, labs, and topics across Lattice.
This study systematically evaluates over 20 token embedding initialization strategies for extending the vocabulary of large language models (LLMs), specifically focusing on Hindi in Nemotron-3-Nano-30B-A3B. The findings reveal that subword composition methods, particularly those utilizing asymmetric variants, significantly outperform traditional vocabulary averaging and other initialization techniques, achieving lower early validation loss and faster convergence. The optimal configuration combines uniform subword averaging with Hindi-specific norm calibration for input embeddings and character-length-weighted subword averaging for the output, resulting in a 6x reduction in continued pre-training steps while maintaining competitive accuracy.
Subword composition methods can drastically reduce continued pre-training steps while enhancing accuracy in LLM vocabulary extensions.
Vocabulary extension is an efficient way to adapt pretrained large language models (LLMs) to new languages, but the initialization of newly added token embeddings can strongly affect continued pre-training (CPT) efficiency. We present a systematic study of more than 20 initialization strategies for Hindi vocabulary extension in Nemotron-3-Nano-30B-A3B. Our comparison spans vocabulary-averaging baselines; external and learned initialization methods, including FOCUS, top-k semantic retrieval, and residual MLP mappings; subword composition; norm calibration; and input-output asymmetry. We find that subword composition methods outperform both vocabulary averaging and external/learned initialization approaches. Within subword composition, asymmetric variants achieve the lowest observed early validation loss and reveal distinct preferences for input and output embedding initialization. The best observed configuration initializes the input embedding matrix with uniform subword averaging and Hindi-specific norm calibration, and the output language modeling head with character-length-weighted subword averaging. Relative to the standard Mean-all baseline, this full initialization pipeline reaches comparable validation loss with over a 6x reduction in CPT steps and exceeds the baseline's 3,500-step MILU-Hindi accuracy after only 500 steps. Finally, we show that initialization loss and initialization bits-per-byte (Init BPB) are unreliable predictors of downstream convergence, whereas lightweight CPT, as few as 50 steps, provides a cost-effective and reliable signal for selecting the best initialization strategy.
