Search papers, labs, and topics across Lattice.
This paper introduces a syllable-level unsupervised speech recognition (UASR) framework that leverages masked language modeling to eliminate the reliance on grapheme-to-phoneme converters and mitigate training instability associated with GAN-based methods. The proposed approach demonstrates a significant improvement, achieving up to a 40% relative reduction in character error rate (CER) on the LibriSpeech dataset. Additionally, it effectively generalizes to low-resource languages, addressing a critical gap in existing speech recognition technologies.
Achieving a 40% reduction in character error rate, this syllable-level UASR framework unlocks new potential for low-resource language recognition without costly phoneme resources.
Training speech recognizers with unpaired speech and text -- known as unsupervised speech recognition (UASR) -- is a crucial step toward extending ASR to low-resource languages in the long-tail distribution and enabling multimodal learning from non-parallel data. However, existing approaches based on phones often rely on costly resources such as grapheme-to-phoneme converters (G2Ps) and struggle to generalize to languages with ambiguous phoneme boundaries due to training instability. In this paper, we address both challenges by introducing a syllable-level UASR framework based on masked language modeling, which avoids the need for G2P and the instability of GAN-based methods. Our approach achieves up to a 40\% relative reduction in character error rate (CER) on LibriSpeech and generalizes effectively to low-resource languages that have remained particularly difficult for prior methods. Code is publicly available\footnote{https://github.com/cactuswiththoughts/SylCipher}.
MIT CSAIL