Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study addresses the high Word Error Rates (WER) of multilingual ASR models like Whisper for Dravidian languages by analyzing linguistic features and dataset characteristics that contribute to decoder inconsistencies. The authors identify that longer words, higher vocabulary diversity, and lower repetition in Dravidian languages lead to sparse token distributions and frequent character-level errors, compounded by an imbalance in attention mechanisms during decoding. By introducing Weighted-Attention and Self-Conditioning techniques, the researchers achieve significant reductions in WER, demonstrating improved performance for low-resource and agglutinative languages.
Dravidian languages can achieve up to 30% lower WER with novel decoder enhancements that balance attention sources and reinforce token predictions.
Multilingual ASR models such as Whisper perform well on high-resource languages but exhibit substantially higher Word Error Rates (WER) for Dravidian languages compared to Indo-Aryan ones. Through linguistic and dataset analysis, we show that Dravidian languages have longer words, higher vocabulary diversity, and lower repetition, resulting in sparse token distributions and frequent character-level substitution errors. Baseline fine-tuning further reveals decoder imbalance between self-attention (linguistic context) and cross-attention (acoustic cues). Although synthetic token-repetition experiments indicate potential gains, they are impractical. Motivated by these observations, we introduce two decoder-level enhancements: Weighted-Attention, which adaptively balances attention sources, and Self-Conditioning, which reinjects intermediate predictions to improve token consistency. Experiments demonstrate consistent WER reductions for low-resource and agglutinative languages.