Search papers, labs, and topics across Lattice.
This paper reevaluates the relationship between language model perplexity (PPL) and automatic speech recognition (ASR) word error rate (WER) in the context of modern end-to-end ASR systems, which incorporate internal language modeling capabilities. The authors find that the traditional linear relationship between PPL and WER does not hold due to the influence of internal language models and the varying effects of encoder context length. Additionally, they demonstrate that subtracting the internal language model from the perplexity calculation alters the PPL-WER relationship, highlighting the need to account for internal language modeling when assessing external language model contributions.
The traditional linear correlation between language model perplexity and ASR word error rate breaks down in modern end-to-end systems, revealing the critical role of internal language modeling.
Language model (LM) perplexity (PPL) has historically been used as a proxy for automatic speech recognition (ASR) word error rate (WER), with prior work reporting an approximately linear relation in log-log space. Modern end-to-end ASR systems challenge this assumption because they already contain internal language modeling capacity, are often evaluated without external language models, and can now be combined with neural LMs and large language models (LLMs) through different recognition strategies. This paper revisits the relation between PPL and WER for modern ASR systems. We study whether external LMs still improve current end-to-end ASR systems, whether the PPL-WER relation remains linear in log-log space, how encoder context length affects this relation, and how LLM perplexities fit into the trend observed for standard neural LMs. We further investigate internal language modeling (ILM) in attention-based encoder-decoder systems and show that ILM subtraction changes the observed PPL-WER relation, indicating that the decoder's internal LM must be considered when interpreting the effect of external LM quality.
