Search papers, labs, and topics across Lattice.
This paper investigates model merging as a lightweight method for adapting instruction-tuned LLMs to low-resource languages, avoiding the need for extensive pre-training or instruction data. They merge an instruction-tuned model with language-specific base models for four Iberian languages (Basque, Catalan, Galician, and Spanish). Results show that merging effectively transfers instruction following behavior and enables multilingual capabilities, offering a computationally efficient alternative to fine-tuning.
Forget fine-tuning: merging language-specific weights into instruction-tuned LLMs unlocks surprisingly effective instruction following in low-resource languages.
Large Language Models (LLMs) remain heavily centered on English, with limited performance in low-resource languages. Existing adaptation approaches, such as continual pre-training, demand significant computational resources. In the case of instructed models, high-quality instruction data is also required, both of which are often inaccessible for low-resource language communities. Under these constraints, model merging offers a lightweight alternative, but its potential in low-resource contexts has not been systematically explored. In this work, we explore whether it is possible to transfer language knowledge to an instruction-tuned LLM by merging it with a language-specific base model, thereby eliminating the need of language-specific instructions and repeated fine-tuning processes whenever stronger instructed variants become available. Through experiments covering four Iberian languages (Basque, Catalan, Galician, and Spanish) and two model families, we show that merging enables effective instruction following behavior in new languages and even supports multilingual capability through the combination of multiple language-specific models. Our results indicate that model merging is a viable and efficient alternative to traditional adaptation methods for low-resource languages, achieving competitive performance while greatly reducing computational cost.
