Search papers, labs, and topics across Lattice.
This paper introduces Flowley, an innovative end-to-end architecture for video-to-audio generation that synthesizes audio directly from visual features and textual prompts, eliminating the need for multi-stage training and reducing computational costs. The authors also present SoundCap, a pipeline for generating detailed sound-aware captions that enhance audio quality by providing better guidance for the model. Flowley achieves state-of-the-art results on the VGGSound benchmark, outperforming existing methods, particularly in zero-shot scenarios, without relying on pretrained audio-visual alignment modules.
Flowley not only streamlines video-to-audio generation with a single-stage architecture but also sets a new benchmark for audio quality by leveraging sound-aware captions.
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.
NVIDIA