Search papers, labs, and topics across Lattice.
This paper introduces CoSync-DiT, a flow matching-based framework for movie dubbing that uses a diffusion transformer architecture to generate speech synchronized with lip movements while preserving vocal identity. The model incorporates acoustic style adaptation, visual calibration, and time-aware context alignment to guide the generative process. A Joint Semantic and Alignment Regularization (JSAR) mechanism is introduced to enforce temporal and semantic consistency, leading to state-of-the-art performance on standard and in-the-wild benchmarks.
Achieve significantly more realistic and lip-synced movie dubbing by modeling the cognitive processes of professional actors with a novel diffusion transformer architecture.
Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics.
