Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FacialTalker, a novel conversational speech synthesis framework that integrates facial expression modeling to enhance empathetic speech generation. By utilizing AUTokenizer, a visual tokenizer that converts facial expressions into compact tokens, and a dual direct preference optimization strategy, the model improves the alignment between visual and speech modalities. Extensive experiments show that FacialTalker significantly outperforms existing methods in both facial expression perception and speech synthesis quality, demonstrating its effectiveness in generating more natural and contextually appropriate speech.
FacialTalker achieves a breakthrough in conversational AI by seamlessly integrating facial expression modeling, leading to speech that resonates more deeply with human emotions.
Conversational Speech Synthesis is a fundamental component of human-computer interaction, aiming to generate contextually appropriate, expressive, and empathetic speech. However, facial expressions encode subtle and rich affective cues that are crucial for empathetic speech interaction, whereas existing approaches often overlook this important modality. In addition, the lack of large-scale natural conversational datasets with both speech and visual modalities also limits the development of visual affect understanding in conversational settings.To address these limitations, we propose FacialTalker, a facial-expression-aware CSS framework built upon a large language model backbone. To efficiently encode facial expressions, we propose AUTokenizer, a single-codebook visual tokenizer that discretizes each frame-level facial expression into a compact token, trained with supervision from combinations of facial Action Units. We further introduce a dual direct preference optimization (DualDPO) strategy, which extends the DPO by jointly imposing preference constraints on both visual and speech token sequences, to enhance the model's understanding of facial expressions and speech semantics in multimodal conversational contexts. Moreover, we construct VSDD-1K, a large-scale multimodal dialogue dataset collected through a fully automated pipeline from real-world Internet conversations, comprising over 1,033 hours of synchronized speaker videos and speech, with more than 85\% of frames containing valid faces. Extensive objective and subjective experiments demonstrate that FacialTalker consistently outperforms strong baselines in facial-expression perception and speech synthesis quality, generating speech that is more natural, expressive, and better aligned with the conversational context. The results also validate the effectiveness of our training strategy and dataset construction pipeline.