Search papers, labs, and topics across Lattice.
This paper introduces HiCoDiT, a Hierarchical Codec Diffusion Transformer designed to enhance video-to-speech (VTS) generation by leveraging the hierarchical structure of speech tokens. By employing low-level blocks that focus on speaker-aware content and high-level blocks that modulate prosodic dynamics, the model achieves improved audio-visual alignment. Extensive experiments show that HiCoDiT significantly outperforms existing methods in both fidelity and expressiveness, demonstrating the effectiveness of discrete modeling in VTS applications.
HiCoDiT achieves superior audio-visual alignment by harnessing the hierarchical nature of speech tokens, outperforming traditional VTS methods in both fidelity and expressiveness.
Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalization and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT.
