Search papers, labs, and topics across Lattice.
The paper introduces HATS, a new French dataset of human-annotated ASR transcriptions where 143 participants chose the better transcription from pairs generated by different ASR systems. The study then investigates the correlation between these human preferences and various ASR evaluation metrics, including WER and embedding-based metrics like BERTscore. Results show that existing metrics, even embedding-based ones, correlate poorly with human perception of transcription quality, highlighting a gap in current ASR evaluation practices.
Current ASR metrics, even those leveraging embeddings, fail to align with human perception of transcription quality, as revealed by a new human-annotated dataset.
Conventionally, Automatic Speech Recognition (ASR) systems are evaluated on their ability to correctly recognize each word contained in a speech signal. In this context, the word error rate (WER) metric is the reference for evaluating speech transcripts. Several studies have shown that this measure is too limited to correctly evaluate an ASR system, which has led to the proposal of other variants of metrics (weighted WER, BERTscore, semantic distance, etc.). However, they remain system-oriented, even when transcripts are intended for humans. In this paper, we firstly present Human Assessed Transcription Side-by-side (HATS), an original French manually annotated data set in terms of human perception of transcription errors produced by various ASR systems. 143 humans were asked to choose the best automatic transcription out of two hypotheses. We investigated the relationship between human preferences and various ASR evaluation metrics, including lexical and embedding-based ones, the latter being those that correlate supposedly the most with human perception.
