Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
The paper introduces SDiaReward, an end-to-end multi-turn reward model for spoken dialogue systems that directly processes speech episodes and is trained on a new dataset of preference pairs. SDiaReward addresses the modality gap (prosody and emotion) and colloquialness gap (natural speech vs. written scripts) in spoken dialogue evaluation. Experiments on the newly created ESDR-Bench demonstrate that SDiaReward outperforms general-purpose audio LLMs in pairwise preference accuracy and captures conversational expressiveness, generalizing across domains and recording conditions.
Current reward models for spoken dialogue systems are missing crucial paralinguistic and natural speech elements, but this new model closes the gap by operating directly on speech and outperforming existing audio LLMs.
The rapid evolution of end-to-end spoken dialogue systems demands transcending mere textual semantics to incorporate paralinguistic nuances and the spontaneous nature of human conversation. However, current methods struggle with two critical gaps: the modality gap, involving prosody and emotion, and the colloquialness gap, distinguishing written scripts from natural speech. To address these challenges, we introduce SDiaReward, an end-to-end multi-turn reward model trained on SDiaReward-Dataset, a novel collection of episode-level preference pairs explicitly targeting these gaps. It operates directly on full multi-turn speech episodes and is optimized with pairwise preference supervision, enabling joint assessment of modality and colloquialness in a single evaluator. We further establish ESDR-Bench, a stratified benchmark for robust episode-level evaluation. Experiments demonstrate that SDiaReward achieves state-of-the-art pairwise preference accuracy, significantly outperforming general-purpose audio LLMs. Further analysis suggests that SDiaReward captures relative conversational expressiveness beyond superficial synthesis cues, improving generalization across domains and recording conditions. Code, data, and demos are available at https://sdiareward.github.io/.