Search papers, labs, and topics across Lattice.
This paper addresses the challenge of speech-preserving facial expression manipulation (SPFEM) by proposing Personalized Cross-Modal Emotional Correlation Learning (PCMECL) to refine VLM-based supervision in the absence of paired training data. PCMECL personalizes VLM prompts using individual visual information to capture fine-grained visual-semantic correlations and employs feature differencing to correlate visual and semantic modalities, bridging the modality gap. Experiments demonstrate that PCMECL significantly improves the performance of existing SPFEM models across various datasets.
Overcome the scarcity of paired data in speech-preserving facial expression manipulation by personalizing visual-language model prompts with individual visual information and correlating changes in visual and semantic features.
Speech-preserving facial expression manipulation (SPFEM) aims to enhance human expressiveness without altering mouth movements tied to the original speech. A primary challenge in this domain is the scarcity of paired data, namely aligned frames of the same individual with identical speech but different expressions, which impedes direct supervision for emotional manipulation. While current Visual-Language Models (VLMs) can extract aligned visual and semantic features, making them a promising source of supervision, their direct application is limited. To this end, we propose a Personalized Cross-Modal Emotional Correlation Learning (PCMECL) algorithm that refines VLM-based supervision through two major improvements. First, standard VLMs rely on a single generic prompt for each emotion, failing to capture expressive variations among individuals. PCMECL addresses this limitation by conditioning on individual visual information to learn personalized prompts, thereby establishing more fine-grained visual-semantic correlations. Second, even with personalization, inherent discrepancies persist between the visual and semantic feature distributions. To bridge this modality gap, PCMECL employs feature differencing to correlate the modalities, providing more precisely aligned supervision by matching the change in visual features to the change in semantic features. As a plug-and-play module, PCMECL can be seamlessly integrated into existing SPFEM models. Extensive experiments across various datasets demonstrate the superior efficacy of our algorithm.
