Search papers, labs, and topics across Lattice.
This paper introduces NVMOS, a novel model for assessing the perceptual quality of non-verbal vocalizations (NVs) in speech, addressing a significant gap in existing speech quality evaluation methods that overlook the quality of NV events themselves. By constructing an NV-MOS dataset from various NV-TTS systems and natural samples, the authors collected expert ratings to benchmark NV quality, revealing inconsistencies in assessments made by multimodal large language models like Gemini. NVMOS achieves expert-level agreement with human judgments, demonstrating its effectiveness in evaluating NV quality in a way that general-purpose models cannot.
General-purpose multimodal models struggle with non-verbal vocalization quality, but NVMOS achieves expert-level accuracy in assessing these crucial acoustic cues.
Non-verbal vocalizations (NVs), such as laughter, sighs, and coughs, are important acoustic cues for emotion and intent. Existing speech quality assessment methods typically focus on overall naturalness, while non-verbal TTS evaluations mainly examine whether a target NV appears with the correct type and position. However, the perceptual quality of NV events themselves remains underexplored. To address this gap, we construct an NV-MOS dataset containing outputs from multiple NV-TTS systems and naturally occurring NV samples, with ratings collected from three acoustic experts on a perceptual quality scale. We further analyze audio-capable multimodal large language models such as Gemini and find clear inconsistencies between their scores and expert ratings. These results suggest that general-purpose multimodal models cannot reliably replace human judgments for NV quality assessment. We then propose NVMOS, to our knowledge the first model that can reliably predict the perceptual quality of NV events in speech. Experimental results show that, with a local NV-event focusing module, NVMOS reaches expert-level or stronger agreement with human MOS.
