Search papers, labs, and topics across Lattice.
This paper investigates the efficacy of merging monolingually pre-trained models to achieve multilingual capabilities, comparing it against mixed-data pre-training. The study reveals that merging monolingual models leads to performance degradation due to representational interference, contrasting with the effectiveness of mixed pre-training. The key finding is that representational similarity is crucial for successful model merging, a condition not met by independently pre-trained monolingual models.
Forget merging monolingual models for multilingual superpowers – this study reveals it's a recipe for performance collapse due to representational interference.
Endowing models with consistent multilingual performance can be achieved by mixing pre-training data, or post-training approaches such as language-specific model merging. In this work, we test whether merging can be applied to monolingually pre-trained models. We conduct a controlled study on the efficacy of mixed, merged, and monolingual pre-training setups. We find that while monolingual pre-training results in strong in-language performance, merging any combination of monolingual models leads to performance collapse due to interference. Our analysis suggests representational similarity is a prerequisite for model merging. We therefore conclude that the flexibility of merging in fine-tuning does not extend trivially to language-specific pre-training.
