Search papers, labs, and topics across Lattice.
This paper addresses the limitations of traditional emotional text-to-speech (TTS) systems that typically model a single emotion per utterance by introducing a novel framework called HybridEmo, which enables multi-emotion control through two tasks: emotion trajectory and emotion blending. By employing Group Relative Policy Optimization and a sample-aware hybrid reward, HybridEmo effectively aligns the speech-token policy to enhance both the accuracy of emotional trajectories and the intensity of blended emotions. The results demonstrate significant improvements in trajectory correctness and blending intensity on the MultiEmo-Test dataset, while maintaining speaker similarity, as confirmed by human evaluations favoring HybridEmo over existing systems.
Multi-emotion TTS can now achieve unprecedented accuracy and expressiveness, outperforming leading models in both trajectory and blending tasks.
Natural-language instructions enable flexible control of synthesized speech, yet emotional TTS systems primarily model a single utterance-level affect, leaving multi-emotion control underexplored. We study two complementary multi-emotion TTS tasks: emotion trajectory, which spans several ordered affective stages, and emotion blending, in which multiple emotions coexist throughout an utterance. These tasks expose a supervision mismatch: supervised fine-tuning (SFT) does not explicitly evaluate emotion features, while single-emotion rewards provide neither structure-aware feedback for trajectory completion nor pair-aware feedback for blending. We introduce HybridEmo, a post-training framework that initializes both tasks with SFT and then aligns the speech-token policy through Group Relative Policy Optimization using a sample-aware hybrid reward. For trajectory samples, segment-aligned consistency combines average and weakest-stage evidence to preserve the correctness and completeness of prescribed stages. For blending samples, a GMM-based reward combines frame-level support from the union of target-emotion anchors in an offline emotion space with an utterance-level weaker-target margin. Both branches share an ASR reward and are routed within a unified policy. On MultiEmo-Test, HybridEmo significantly improves trajectory correctness and blending intensity, without a noticeable degradation in speaker similarity. Human evaluation prefers HybridEmo to CosyVoice 3 and EmoVoice-0.5B, with nearly balanced preferences against Qwen3-TTS.
