Search papers, labs, and topics across Lattice.
This study systematically audits the performance of eight state-of-the-art Automatic Speech Recognition (ASR) models on real-world psychiatric interview data in Kannada, Hindi, and Indian English, revealing significant performance disparities across languages and speaker demographics. The authors fine-tune the best-performing models, Gemma3n and OmniLingual, and identify systematic gaps in ASR accuracy related to speaker role and gender, highlighting potential biases in clinical applications. To address these issues, they introduce SamaVaani, a unified debiasing technique that enhances both ASR performance and fairness across demographic groups.
Systematic audits reveal that ASR models can significantly underperform for regional languages and specific demographics, raising critical equity concerns in clinical settings.
Automatic Speech Recognition (ASR) is increasingly used to document clinical encounters, yet its reliability in multilingual and demographically diverse Indian healthcare context remains largely unknown. In this study, we first conduct the systematic audit of ASR performance on real-world psychiatric interview data spanning Kannada, Hindi and Indian English, comparing eight state-of-the-art models including IndicWhisper, WhisperLargeV3, Sarvam, GoogleS2T, Gemma3n, OmniLingual, Vaani, and Gemini. Our results reveal substantial variability across models and languages, with some systems performing competitively in Indian English but failing in regional speech. We further fine-tune two of the best performing opensource models, i.e., Gemma3n and OmniLingual, using various methods. With this, we uncover systematic performance gaps tied to speaker role and gender, raising concerns about equitable deployment in clinical settings, which are further mitigated by fairness-aware fine-tuning. To this end, we propose SamaVaani, a unified debiasing technique that simultaneously improves ASR performance and improves fairness across demographic groups.
