Search papers, labs, and topics across Lattice.
This paper introduces ReFree-S2V, a novel framework for generating realistic speech-driven portrait animations that effectively balances lip synchronization and expressive behavior. By utilizing a multi-level speech representation that captures both phonetic and prosodic features, the model selectively integrates these cues into transformer blocks, allowing for enhanced articulation and natural motion. The introduction of a reward-free reinforcement learning approach further refines the training process, leading to significant improvements in both lip-sync accuracy and overall animation quality compared to existing methods.
Achieving state-of-the-art lip-sync accuracy while maintaining natural expressiveness, ReFree-S2V redefines the standards for speech-driven character animation.
Speech-driven talking character animation seeks to generate life-like portrait videos that convey natural conversation behavior, aligning facial motion with spoken audio. Although recent advances in video generation have substantially improved realism in video-based animation, achieving both accurate lip articulation and expressive behavior remains challenging. Existing approaches typically trade off precise phoneme-to-lip synchronization against dynamic facial expressions and head motion, yielding animations that are either accurate yet rigid, or expressive but poorly synchronized. We address this challenge by proposing ReFree-S2V, a flow-matching speech-to-portrait animation framework that builds upon a pretrained video generation model to achieve fine-grained speech articulation and high-level expressive cues in speech-driven portrait animation. This model introduces a multi-level speech representation capturing phonetic and prosodic information at both local and global granularities. These representations are selectively injected into transformer blocks via learnable level selectors, enabling both accurate lip synchronization and natural expressive motion. To achieve natural head movements, we further introduce a novel reward-free reinforcement learning scheme into flow-matching training to discourage perceptually implausible motion without relying on handcrafted synchronization metrics or reward models, or the high cost of human preference annotation. Extensive experiments demonstrate that ReFree-S2V achieves state-of-the-art performance, significantly outperforming existing methods in both quantitative lip-sync accuracy and qualitative human evaluations of naturalness and expressivity.
