Search papers, labs, and topics across Lattice.
This paper introduces a fine-tuning-free talking face generation framework leveraging Stable Diffusion and IP-Adapter to reduce computational costs and improve accessibility. The method incorporates three novel trainable-parameter-free components: a Structurist for disentangling lip and appearance features, a Structure Controller for lip synchronization, and a Noise Sensor for temporal consistency. Experiments demonstrate state-of-the-art performance in lip-sync accuracy (PCLD) and visual fidelity (FID) compared to existing fine-tuned approaches.
Achieve state-of-the-art talking face generation without any fine-tuning, proving that pre-trained diffusion models like Stable Diffusion already possess strong lip-related semantics.
With the rapid advancement of diffusion models, talking face generation has made remarkable progress. However, existing diffusion-based methods still require task-specific fine-tuning and large-scale audiovisual datasets, resulting in high computational costs that hinder scalability and accessibility of diffusion-based approaches across the research community. To address this, we propose a finetuning-free paradigm that directly performs talking face generation using the pretrained weights of Stable Diffusion and IP-Adapter. This backbone leverages the visual embedding capability of IP-Adapter to mine lip-related semantics from the pretrained Stable Diffusion. To address the challenges of identity drift, synchronization errors, and temporal instability, we also design three trainable-parameterfree components: (1) the Structurist, which explicitly disentangles and reassembles lip and appearance features to mitigate identity drift and appearance distortion; (2) the Structure Controller, which adaptively refines embeddings based on quasi-monotonic motion trends for precise lip synchronization; and (3) the Noise Sensor, which introduces Gaussian prior to detect and suppress flicker and jitter artifacts and enhance temporal consistency. Experimental results show that our method outperforms existing SOTA approaches in both lip-sync accuracy (at least 0.16 gain in PCLD) and visual fidelity (at least 0.7 improvement in FID), establishing a novel fine-tuning-free diffusion framework for talking face generation.
