Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates the performance of five reference-based objective intelligibility metrics (OIMs) and two deep learning-based no-reference metrics on a newly developed Spanish speech intelligibility dataset, SpInt. The findings reveal that reference-based metrics significantly outperform no-reference approaches, particularly when faced with training-test acoustic mismatches, such as language differences. To support future research in this area, the authors have made the SpInt dataset publicly available, highlighting the need for more robust no-reference OIMs that can generalize across languages.
Reference-based metrics dominate no-reference approaches in evaluating Spanish speech intelligibility, revealing critical gaps in current deep learning methods.
Objective intelligibility metrics (OIMs) enable fast and low-cost evaluation of speech intelligibility and are widely used in speech technology assessment. In this study, we evaluate five reference-based OIMs (STOI, ESTOI, STGI, HASPI, and SIIB) and two deep learning-based no-reference metrics (MOSA-Net+ and W2V-SIP) on SpInt, a new Spanish speech intelligibility dataset. Our results show that reference-based OIMs consistently outperform modern data-driven no-reference approaches, which degrade notably under training-test acoustic mismatches such as language mismatch. This effect is particularly relevant in our scenario, as none of the evaluated metrics were exposed to Spanish speech data during development. Consequently, to foster research on more robust and generalizable no-reference OIMs, SpInt is released publicly.