Search papers, labs, and topics across Lattice.
HoliDubber introduces a novel framework for video dubbing that synthesizes both speech and sound effects from a single text prompt, overcoming the limitations of traditional systems that focus solely on speech synthesis. By employing a patch-based autoregressive diffusion transformer architecture, the model captures global temporal structures and achieves high-fidelity audio generation through a divide-and-conquer strategy. Extensive evaluations demonstrate that HoliDubber significantly surpasses existing methods in speech quality, synchronization, and speaker similarity, establishing a new standard for holistic dubbing in complex acoustic environments.
HoliDubber revolutionizes video dubbing by seamlessly integrating speech and sound effects from a single text prompt, outperforming traditional methods in quality and synchronization.
Video dubbing is a cornerstone of multimedia content creation, aiming to synthesize synchronized acoustic sequences for visual streams. While Text-to-Speech (TTS) and Text-to-Audio (TTA) generation have each achieved remarkable progress, existing dubbing systems remain confined to isolated speech synthesis without incorporating sound effects and ambient audio, forcing practitioners to rely on fragmented workflows and laborious manual post-mixing. To address this limitation, we present HoliDubber, a holistic video dubbing framework that moves beyond speech-only generation by enabling the joint synthesis of speech and sound effects from a single text prompt. Specifically, HoliDubber adopts a patch-based autoregressive diffusion transformer architecture, where a causal language model autoregressively models aggregated patch embeddings to capture global temporal structure, and a Diffusion Transformer decoder generates high-fidelity continuous tokens within each patch, following a divide-and-conquer strategy. To achieve cross-modal alignment, visual features are encoded into patch-level representations and fused with audio patches via cross-attention, enabling the model to ground speech generation in the speaker's visual articulation dynamics. In addition, we introduce HoliDub-Bench, a benchmark curated from established datasets with synchronized video-text-audio triplets designed for holistic dubbing evaluation. Extensive experiments demonstrate that HoliDubber significantly outperforms existing methods across multiple benchmarks in speech quality, synchronization, and speaker similarity. Furthermore, results on HoliDub-Bench validate the effectiveness of joint speech-and-sound generation, establishing a new paradigm for holistic video dubbing in complex acoustic scenes. \footnote{The demo page of the project is https://holidubber.github.io}
