Search papers, labs, and topics across Lattice.
The paper addresses the problem of pessimistic ASR evaluation for Indian languages due to orthographic variations by introducing Orthographically-Informed Word Error Rate (OIWER). They leverage LLMs to create benchmarks that capture permissible orthographic variations in Indian languages. Experiments demonstrate that OIWER reduces pessimistic error rates, narrows inflated model gaps, and aligns more closely with human perception compared to WER and WER-SN.
LLMs can help you build better ASR benchmarks for low-resource languages, reducing error rates by 6.3 points and better aligning with human perception.
Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points.
