Search papers, labs, and topics across Lattice.
This study investigates the ability of large audio-language models (LALMs) to make comparative judgments about emotions in speech signals, focusing on dimensions such as arousal, valence, and dominance. The authors introduce a reasoning-guided ordinal speech emotion recognition (SER) framework that conditions the LALM on paired speech inputs, utilizing reasoning traces from semantic audio descriptions and GeMAPS acoustic features. Results indicate that this framework enhances preference prediction accuracy while significantly reducing the training data requirement to just 5% of that needed by traditional ordinal SER systems.
LALMs can achieve superior emotional comparison accuracy with only 5% of the training data typically required by conventional methods.
Large audio-language models (LALMs) can reason about audio, yet it remains unclear whether they can perform comparative judgments between two speech signals along emotional, environmental, linguistic, prosodic, and interpersonal dimensions. We study this question in the context of speech emotion recognition (SER), where the model determines which utterance exhibits higher arousal, valence, or dominance. We introduce a reasoning-guided ordinal SER framework that conditions an LALM on paired speech inputs. The model is trained using reasoning traces generated from both semantic audio descriptions and acoustic evidence derived from GeMAPS features, enabling interpretable comparative decisions. Beyond direct supervision, we also employ direct preference optimization to encourage stronger separation for emotional differences. Experiments show that the proposed framework improves preference prediction while requiring only 5% of the training data used by conventional ordinal SER systems.
CMU ML
NVIDIA