Search papers, labs, and topics across Lattice.
This study evaluates the performance of two phoneme-based automatic speech recognition (ASR) systems, WhisperIPA and ZIPA, focusing on their ability to generate International Phonetic Alphabet (IPA) transcriptions across various accents and demographic groups. By comparing these models against grapheme-to-phoneme systems using standard phoneme error rate (PER) and a new Soft PER metric, the research uncovers significant performance disparities related to gender, accent, ethnicity, and age, even when accounting for acceptable phonemic variations. These insights highlight the persistent biases in ASR systems and underscore the need for more inclusive approaches in phoneme-based ASR development.
Despite advancements in ASR technology, significant demographic biases persist in phoneme-based systems, revealing critical gaps in their performance across diverse user groups.
The popularization of automatic speech recognition (ASR) systems has increased exploration of the demographic biases related to race, age, gender, and accent, often formed from imbalanced training data. Most of these studies focused on standard grapheme-based ASR systems with comparatively little emphasis on phoneme-based systems, such as models that produce International Phonetic Alphabet (IPA) representations. As ASR systems shift toward multilingual support and low-resource language modeling, IPA-based layers serve as a critical, language-agnostic foundation. In this study, we evaluate the performance of two state-of-the-art open-source ASR systems, WhisperIPA and ZIPA, that generate IPA transcriptions across diverse accents and language sources. Our evaluation includes existing multilingual speech corpora and demographically annotated English-language corpora. We measure model performance by comparing model-generated IPA transcriptions against grapheme-to-phoneme (G2P) systems using both standard phoneme error rate (PER) and a proposed Soft PER metric that tolerates linguistically similar phoneme substitutions. Our analysis examines how performance varies across languages and demographic groups such as gender, accent, ethnicity, and age, revealing persistent disparities even after accounting for acceptable phonemic variation. These findings provide insight into potential sources of bias and inform the development of more inclusive and linguistically robust phoneme-based ASR systems. Our code and data will be made publicly available to the community.
