Search papers, labs, and topics across Lattice.
The paper introduces SDiaReward, an end-to-end multi-turn reward model for spoken dialogue systems that explicitly addresses the modality gap (prosody and emotion) and the colloquialness gap (natural speech vs. written scripts). SDiaReward is trained on the SDiaReward-Dataset, a new dataset of episode-level preference pairs, and optimized with pairwise preference supervision to jointly assess modality and colloquialness. Experiments on the newly established ESDR-Bench demonstrate that SDiaReward outperforms general-purpose audio LLMs in pairwise preference accuracy and captures conversational expressiveness, improving generalization.
Spoken dialogue systems can now be better optimized for naturalness and expressiveness thanks to a new reward model that understands the nuances of prosody, emotion, and colloquial speech.
The rapid evolution of end-to-end spoken dialogue systems demands transcending mere textual semantics to incorporate paralinguistic nuances and the spontaneous nature of human conversation. However, current methods struggle with two critical gaps: the modality gap, involving prosody and emotion, and the colloquialness gap, distinguishing written scripts from natural speech. To address these challenges, we introduce SDiaReward, an end-to-end multi-turn reward model trained on SDiaReward-Dataset, a novel collection of episode-level preference pairs explicitly targeting these gaps. It operates directly on full multi-turn speech episodes and is optimized with pairwise preference supervision, enabling joint assessment of modality and colloquialness in a single evaluator. We further establish ESDR-Bench, a stratified benchmark for robust episode-level evaluation. Experiments demonstrate that SDiaReward achieves state-of-the-art pairwise preference accuracy, significantly outperforming general-purpose audio LLMs. Further analysis suggests that SDiaReward captures relative conversational expressiveness beyond superficial synthesis cues, improving generalization across domains and recording conditions. Code, data, and demos are available at https://sdiareward.github.io/.
