Search papers, labs, and topics across Lattice.
Interactive TTS tackles timbre drift and poor style control in multi-turn conversational speech synthesis by decoupling implicit context modeling into explicit, executable style instructions. The framework leverages Iterative Rejection Sampling Fine-Tuning (Iterative RSFT) alongside Context-Aware Direct Preference Optimization (CADPO) to tightly couple generated acoustic styles with dialogue history. Consequently, the model establishes state-of-the-art expressive synthesis on VStyle and SpeechParaling-Bench while preserving cross-turn speaker identity.
Timbre drift in multi-turn expressive voice agents can be eliminated by replacing implicit context embeddings with explicitly supervised style instructions optimized via rejection sampling and context-aware DPO.
Dynamic speaking style adaptation in multi-turn multimodal interaction remains a major challenge for text-to-speech (TTS) systems. Existing context-aware TTS (CTTS) methods typically map dialogue context to speech in an end-to-end manner. Such implicit modeling makes contextual style decisions difficult to supervise, while the entanglement of style, timbre, and content often leads to weak instruction-following and severe timbre drift across turns. To overcome these limitations, we propose Interactive TTS, a dynamic, style-adaptive framework for contextually appropriate and speaker-consistent speech generation. Interactive TTS decouples the process by explicitly modeling contextual style decisions as executable instructions. To bridge the gap between style decisions and speech generation, we introduce Iterative Rejection Sampling Fine-Tuning (Iterative RSFT) and Context-Aware Direct Preference Optimization (CADPO), which significantly enhance instruction-following and align the generated speech with conversational contexts. Extensive experiments demonstrate that Interactive TTS outperforms state-of-the-art models on VStyle and SpeechParaling-Bench. Demo is available at https://wjtian-wonderful.github.io/InteractiveTTS/
