Search papers, labs, and topics across Lattice.
This study introduces MultiClin, a benchmark for evaluating automatic speech recognition (ASR) systems in clinical settings that accounts for multiscript variability, where the same term can appear in different orthographic forms. By demonstrating that traditional string-matching metrics underestimate ASR performance, the authors show that multiscript-aware evaluation provides a more accurate assessment of recognition quality. Additionally, the research reveals that inconsistent script mappings during training lead to increased orthographic uncertainty and hinder model convergence, while script unification enhances ASR performance.
Multiscript-aware evaluation reveals that traditional ASR metrics can misrepresent performance by treating valid orthographic variants as errors.
Automatic speech recognition (ASR) in non-English clinical settings is challenged by multiscript variability, where the same term may appear in multiple valid orthographic forms. Conventional string-matching evaluation metrics often underestimate ASR performance by treating orthographic variants as errors. To address this issue, we introduce MultiClin, a clinical ASR benchmark designed to evaluate robustness to multiscript variability. Experiments across diverse ASR models show that multiscript-aware evaluation provides a fairer assessment of recognition quality than conventional single-reference evaluation. We further investigate the impact of script consistency during training and find that inconsistent script mappings increase orthographic uncertainty and hinder model convergence, with a balanced 50% mapping ratio producing the highest entropy. In contrast, script unification consistently yields the best ASR performance. Our dataset and code are publicly available at: https://github.com/aitrics-ronaldo/Interspeech_MultiClin.
