Search papers, labs, and topics across Lattice.
This paper introduces Script-Normalized WER (SN-WER), a novel evaluation metric for automatic speech recognition (ASR) that addresses the overestimation of errors caused by script mismatches in multilingual contexts. By transliterating both reference and hypothesis texts into a canonical script prior to WER computation, SN-WER significantly reduces inflated model gaps by up to 12% on curated datasets, while also revealing genuine recognition weaknesses in noisier datasets. The method demonstrates robustness across various transliteration choices and shows a marked attenuation of artificial WER inflation due to romanization, advocating for its use alongside traditional WER and CER metrics in ASR evaluations.
SN-WER effectively mitigates script-induced error inflation in ASR evaluations, revealing true model performance gaps.
Word Error Rate (WER) is the dominant metric for automatic speech recognition (ASR), but it can overestimate errors when references and hypotheses encode the same words in different scripts. This issue is common in multilingual settings where ASR models may emit romanized text. We propose Script-Normalized WER (SN-WER), a training-free, evaluation-only scoring method that transliterates both reference and hypothesis text into a language-specific canonical script before computing WER. We evaluate SN-WER on 5 Indic languages, 2 datasets, and 3 ASR models. On curated FLEURS data, SN-WER reduces inflated model gaps by up to 12%, while on noisier Common Voice data the reductions are smaller or inconsistent, indicating genuine recognition weaknesses rather than only script mismatch. Controlled stress tests show a 67% attenuation of artificial romanization-induced WER inflation, while lexical-substitution controls show near-identical sensitivity to semantic errors, with Delta SN-WER / Delta WER approximately 1.09. SN-WER is robust to transliterator choice, normalization changes, and shows low token-collision rates below 0.1% in the evaluated Indic setting. We argue that SN-WER should be reported alongside WER and CER as a companion metric for script-insensitive ASR evaluation, especially when transcripts feed downstream search, indexing, or multilingual LLM pipelines.
