Search papers, labs, and topics across Lattice.
This study compares encoder and decoder-based Large Language Models (LLMs) for evaluating Automatic Speech Recognition (ASR) systems, focusing on embedding-based metrics like BERTScore and SemDist. The authors find that both types of models can achieve high correlation with human judgments when optimized, with generative LLMs excelling in hypothesis selection and qualitative error classification. Notably, encoder-based metrics maintain competitive performance, suggesting a nuanced understanding of their roles in ASR evaluation is essential.
Encoder models hold their ground against generative LLMs in ASR evaluation, but the latter enhance interpretability and hypothesis selection.
Automatic Speech Recognition (ASR) is typically evaluated using Word Error Rate (WER), which poorly reflects semantic similarity. While embedding-based metrics correlate better with human judgments, the respective roles of encoder and decoder-based Large Language Models (LLMs) remain underexplored. This paper presents a comparative study of both families for ASR evaluation. We analyze BERTScore and SemDist across different LLMs, layers, and pooling strategies, showing that both metrics can achieve strong correlation with human judgments when properly configured. For decoder models, we investigate generative LLMs in two settings: pairwise hypothesis selection via prompting and direct qualitative error classification. Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.
