Search papers, labs, and topics across Lattice.
This paper introduces DAVE, a novel decoupled audio-visual enhancement framework designed to improve speech separation in challenging real-world conditions. By constructing the DAVE-Corpus, a large-scale dataset with over 219,000 audio-visual mixtures, the authors address the scarcity of training data while employing a progressive multi-objective optimization strategy to enhance speech intelligibility and quality. Experimental results indicate that DAVE outperforms existing methods, demonstrating significant robustness against visual degradation and mixed acoustic scenarios.
DAVE achieves superior speech separation performance even in the presence of unreliable visual inputs, thanks to its innovative decoupled framework and extensive training dataset.
Audio-visual speech enhancement under real-world conditions remains challenging due to unreliable visual inputs and the lack of large-scale training data with realistic acoustic conditions. Existing approaches usually fuse visual features directly into the separation network, making them vulnerable to degraded visual signals. In this paper, we present DAVE, a decoupled audio-visual enhancement framework for real-world speech separation. Firstly, to address the data scarcity issue, we construct DAVE-Corpus, a large-scale training corpus with 219,411 mixtures generated from public meeting corpora through combinatorial acoustic augmentation. Then, we introduce a progressive multi-objective optimization strategy to jointly improve speech separation, intelligibility, speaker identity preservation, and perceptual quality. We further develop a certified selective enhancement chain that applies scene routing, GAN-based denoising, and loudness normalization only within the no-reference partition, guaranteeing non-degradation of reference-based metrics. Experimental results on the Real-World Audio-Visual Speech Enhancement Challenge demonstrate the robustness of DAVE under both real-world mixed scenarios and visual degradation conditions.
