Search papers, labs, and topics across Lattice.
This paper introduces GaussianEmoTalker, an innovative framework for real-time emotional talking head synthesis that leverages 3D Gaussian Splatting to enhance expressiveness and controllability in avatar animations. By treating emotional animation as a residual deformation problem, the method constructs a neutral talking space and predicts emotion-conditioned deformations using a novel spatial-audio-emotion attention module. Extensive evaluations reveal that GaussianEmoTalker achieves superior video quality, precise lip synchronization, and dynamic emotional expression, all while maintaining real-time performance.
Emotional avatars can now be synthesized in real-time with unprecedented expressiveness and control, thanks to a novel approach that combines audio features and Gaussian attributes.
Audio-driven talking head synthesis has achieved impressive progress in lip synchronization and visual quality, yet generating expressive emotional avatars with controllable intensity remains challenging, especially under real-time constraints. In this paper, we present GaussianEmoTalker, an audio-driven framework for real-time emotional talking head synthesis based on 3D Gaussian Splatting. Instead of directly predicting the final emotional avatar from speech, we formulate emotional animation as a neutral-to-emotional residual deformation problem. GaussianEmoTalker first constructs an identity-specific neutral talking space with GaussianBlendshapes, which provides high-fidelity Gaussian attributes and phoneme-synchronized neutral motion. It then predicts an emotion-conditioned residual deformation by combining mesh displacement cues, audio features, emotion categories, and intensity encodings. To fuse these heterogeneous signals, we introduce a spatial-audio-emotion attention module that estimates the offsets of Gaussian attributes for expressive and temporally stable rendering. Extensive experiments demonstrate that GaussianEmoTalker achieves competitive video quality, accurate lip synchronization, controllable emotional expression, and real-time rendering compared with recent emotional talking head methods. Our project page is available at https://njust-yang.github.io/GaussianEmoTalker.github.io/
