Search papers, labs, and topics across Lattice.
The authors introduce VidAudio-Bench, a new benchmark for Video-to-Audio (V2A) and Video-Text-to-Audio (VT2A) generation, covering sound effects, music, speech, and singing. The benchmark includes 1,634 video-text pairs and 13 task-specific, reference-free metrics validated through human studies. Results show current V2A models struggle with speech and singing, and VT2A models face a trade-off between instruction following and visual grounding.
Current video-to-audio models are surprisingly bad at generating speech and singing, exposing critical gaps in multimodal understanding.
Video-to-Audio (V2A) generation is essential for immersive multimedia experiences, yet its evaluation remains underexplored. Existing benchmarks typically assess diverse audio types under a unified protocol, overlooking the fine-grained requirements of distinct audio categories. To address this gap, we propose VidAudio-Bench, a multi-task benchmark for V2A evaluation with four key features: (1) Broad Coverage: It encompasses four representative audio categories - sound effects, music, speech, and singing - under both V2A and Video-Text-to-Audio (VT2A) settings. (2) Extensive Evaluation: It comprises 1,634 video-text pairs and benchmarks 11 state-of-the-art generation models. (3) Comprehensive Metrics: It introduces 13 task-specific, reference-free metrics to systematically assess audio quality, video-audio consistency, and text-audio consistency. (4) Human Alignment: It validates all metrics through subjective studies, demonstrating strong consistency with human preferences. Experimental results reveal that current V2A models perform poorly in speech and singing compared to sound effects. Our VT2A results further highlight a fundamental tension between instruction following and visually grounded generation: stronger visual conditioning improves video-audio alignment, but often at the cost of generating the intended audio category. These findings establish VidAudio-Bench as a comprehensive and scalable framework for diagnosing V2A systems and provide new insights into multimodal audio generation.
