Search papers, labs, and topics across Lattice.
To overcome multilingual capacity dilution and poor tail-language coverage in massive speech models, the authors scaled monolingual adaptation across 102 FLEURS languages by fine-tuning Whisper with specialized tokenizers and text-only data augmentation. The resulting model swarm outperforms Whisper-large-v3 on 77 of 102 languages, reducing character error rates by an average factor of 2.8x and achieving open-source state-of-the-art results on 27 languages. Crucially, dedicated monolingual tokenizers boost sequence compression by an average of 3.3x (and up to 21.7x), dramatically improving decoding efficiency for non-Latin and underrepresented scripts.
Multilingual speech foundations leave massive performance on the table: specialized monolingual Whisper models beat Whisper-large-v3 across 77 of 102 languages while slashing character error rates by nearly 3x.
We introduce BuzzASR, a collection of language-specialized fine-tuned Whisper models adapted for automatic speech recognition (ASR) in 102 languages. Large end-to-end Transformer-based ASR models such as Whisper have revolutionized ASR, but most prominent models are highly multilingual. As a result, these models often perform poorly on languages less well-represented in their training set. While it has long been known that effective language adaptation can be achieved through simple fine-tuning on monolingual data, this strategy has only been applied to a small number of languages. We massively scale up this simple approach to 102 languages covered in the FLEURS dataset, while also implementing a more complex language adaptation strategy that integrates monolingual tokenizer replacement and data augmentation using text-only fine-tuning. BuzzASR models outperform Whisper-large-v3 on 77 out of 102 languages, reducing character error rates (CER) by a factor of over 2.8 on average. Our models achieve state-of-the-art CER among open-source systems on 27 of 102 languages on the combined FLEURS and Common Voice test set. Our tokenizer replacement strategy yields an average 3.3x improvement in compression rate (characters per token) over Whisper's multilingual BPE, with gains of up to 21.7x. We release all models, code, and detailed results: https://lemn-lab.github.io/buzz-asr
