Search papers, labs, and topics across Lattice.
This paper investigates the use of song lyrics as a data source for fine-tuning Kazakh ASR models, addressing the challenge of limited transcribed corpora for low-resource languages. The authors curated a dataset of 3,013 song lyric-audio pairs and fine-tuned Whisper models under various training scenarios combining the song data with Common Voice Corpus (CVC) and FLEURS. Results demonstrate that fine-tuning with song data improves performance over zero-shot Whisper, particularly on the Kazakh Speech Corpus 2 (KSC2) benchmark, suggesting song lyrics can be a viable data augmentation strategy for low-resource ASR.
Who knew that fine-tuning ASR models on song lyrics could halve the error rate for low-resource languages like Kazakh?
Developing automatic speech recognition (ASR) systems for low-resource languages is hindered by the scarcity of transcribed corpora. This proof-of-concept study explores songs as an unconventional yet promising data source for Kazakh ASR. We curate a dataset of 3,013 audio-text pairs (about 4.5 hours) from 195 songs by 36 artists, segmented at the lyric-line level. Using Whisper as the base recogniser, we fine-tune models under seven training scenarios involving Songs, Common Voice Corpus (CVC), and FLEURS, and evaluate them on three benchmarks: CVC, FLEURS, and Kazakh Speech Corpus 2 (KSC2). Results show that song-based fine-tuning improves performance over zero-shot baselines. For instance, Whisper Large-V3 Turbo trained on a mixture of Songs, CVC, and FLEURS achieves 27.6% normalised WER on CVC and 11.8% on FLEURS, while halving the error on KSC2 (39.3% vs. 81.2%) relative to the zero-shot model. Although these gains remain below those of models trained on the 1,100-hour KSC2 corpus, they demonstrate that even modest song-speech mixtures can yield meaningful adaptation improvements in low-resource ASR. The dataset is released on Hugging Face for research purposes under a gated, non-commercial licence.
