Search papers, labs, and topics across Lattice.
This paper introduces Unified Gradient Projection (UGP), a novel approach to mitigate catastrophic forgetting in multilingual automatic speech recognition (ASR) systems when fine-tuning on low-resource languages. By employing a language-balanced replay mechanism that constrains parameter updates through reference gradients in a unified projection space, UGP effectively reduces the bias introduced by dominant languages during optimization. Experimental results demonstrate that UGP achieves near-zero average forgetting across various low-resource language groups and model scales, significantly enhancing cross-lingual stability and adaptability.
UGP enables multilingual ASR models to achieve near-zero forgetting while adapting to low-resource languages, revolutionizing how we approach continual learning in diverse linguistic contexts.
Large-scale pretrained ASR models such as Whisper exhibit strong multilingual capabilities. However, fine-tuning on low-resource languages often causes catastrophic forgetting. Although continual learning mitigates this issue, existing methods struggle to regulate cross-task interference in multilingual settings, where dominant languages bias optimization. We propose Unified Gradient Projection (UGP), which constrains parameter updates using reference gradients from language-balanced replay in a unified projection space. By equalizing per-language contributions in the projection, UGP reduces dominant-language bias and improves cross-lingual stability. We further show that combining gradient-level projection with data-level replay yields complementary gains in stability and plasticity. Across diverse low-resource language groups and model scales, UGP enables effective adaptation while substantially mitigating forgetting. On Whisper-large-v3, it achieves near-zero average forgetting.
Tsinghua AI