Search papers, labs, and topics across Lattice.
This paper introduces TD-V2A, a novel approach to video-to-audio generation that utilizes temporal differences (TD) to enhance visual conditioning without requiring additional networks or complex inductive biases. By investigating TD at both frame and feature levels, the authors identify optimal representation strategies that improve audio synthesis quality. Extensive experiments show that TD-V2A significantly outperforms existing methods, including those relying on contrastive audio-visual pretraining, indicating that temporal cues are crucial for effective V2A generation.
Leveraging temporal differences can dramatically enhance video-to-audio generation quality, outperforming even dedicated multimodal representations.
Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.
Tsinghua AI