Search papers, labs, and topics across Lattice.
This paper introduces and investigates Relative Voice Impression Estimation (RIE), a novel framework for predicting the perceptual difference between two utterances by the same speaker. The authors compare classical acoustic features, self-supervised speech representations, and multimodal large language models (MLLMs) for predicting low-dimensional vectors representing the perceptual shift between utterances along antonymic axes. Results show that self-supervised speech representations outperform classical acoustic features, while MLLMs perform poorly on this fine-grained pairwise task.
Forget MLLMs: self-supervised speech models are surprisingly good at capturing subtle differences in voice impressions, outperforming traditional acoustic features.
Paralinguistic and non-linguistic aspects of speech strongly influence listener impressions. While most research focuses on absolute impression scoring, this study investigates relative voice impression estimation (RIE), a framework for predicting the perceptual difference between two utterances from the same speaker. The estimation target is a low-dimensional vector derived from subjective evaluations, quantifying the perceptual shift of the second utterance relative to the first along an antonymic axis (e.g., ``Dark--Bright''). To isolate expressive and prosodic variation, we used recordings of a professional speaker reading a text in various styles. We compare three modeling approaches: classical acoustic features commonly used for speech emotion recognition, self-supervised speech representations, and multimodal large language models (MLLMs). Our results demonstrate that models using self-supervised representations outperform methods with classical acoustic features, particularly in capturing complex and dynamic impressions (e.g., ``Cold--Warm'') where classical features fail. In contrast, current MLLMs prove unreliable for this fine-grained pairwise task. This study provides the first systematic investigation of RIE and demonstrates the strength of self-supervised speech models in capturing subtle perceptual variations.
