Search papers, labs, and topics across Lattice.
This study systematically investigates the effects of perturbation-based continued pre-training (CPT) on multilingual language models, focusing on dialect robustness across nine tasks in German, Italian, and Arabic. By comparing six training conditions, the research reveals that character-noised CPT significantly enhances zero-shot dialect robustness while maintaining performance on standard varieties. Importantly, the findings indicate that different CPT strategies lead to distinct mechanisms of robustness, influencing language model adaptation and prediction repair in unique ways.
Character-noised continued pre-training boosts zero-shot dialect robustness while preserving standard performance, revealing distinct adaptation mechanisms in language models.
Dialectal variation remains a major challenge for multilingual language models. Perturbation-based continued pre-training (CPT) has emerged as a promising approach to improving robustness, yet existing work largely evaluates individual perturbation strategies in isolation and provides limited insight into why they work. We present a systematic study of perturbation-based CPT for multilingual dialect robustness in LLMs, comparing six training conditions across nine German, Italian, and Arabic dialect tasks. Perturbation-based CPT, especially character-noised CPT, consistently improves zero-shot dialect robustness while largely preserving standard variety performance. More importantly, we show that methods with similar downstream performance induce distinct mechanisms of robustness, exhibiting different patterns of language model adaptation, representational alignment, and prediction repair. Our results provide a more complete understanding of how synthetic surface variation improves robustness and offer practical guidance for selecting CPT strategies in multilingual and dialectal settings.
