Search papers, labs, and topics across Lattice.
This paper introduces a Visual-Informed Neural Beamforming Network (VI-NBFNet) for speech enhancement that leverages both microphone array signals and visual lip movement features extracted from a pretrained visual speech recognition model. The network uses these multimodal inputs for voice activity detection, target speaker identification, and supervised end-to-end beamforming with an attention mechanism. Experiments show that VI-NBFNet achieves superior speech enhancement performance and robustness in both stationary and dynamic speaker scenarios compared to existing methods.
Attention-based beamforming, guided by visual lip movements, unlocks more robust speech enhancement, even with moving speakers and challenging noise conditions.
Recent studies have demonstrated that incorporating auxiliary information, such as speaker voiceprint or visual cues, can substantially improve Speech Enhancement (SE) performance. However, single-channel methods often yield suboptimal results in low signal-to-noise ratio (SNR) conditions, when there is high reverberation, or in complex scenarios involving dynamic speakers, overlapping speech, or non-stationary noise. To address these issues, we propose a novel Visual-Informed Neural Beamforming Network (VI-NBFNet), which integrates microphone array signal processing and deep neural networks (DNNs) using multimodal input features. The proposed network leverages a pretrained visual speech recognition model to extract lip movements as input features, which serve for voice activity detection (VAD) and target speaker identification. The system is intended to handle both static and moving speakers by introducing a supervised end-to-end beamforming framework equipped with an attention mechanism. The experimental results demonstrated that the proposed audiovisual system has achieved better SE performance and robustness for both stationary and dynamic speaker scenarios, compared to several baseline methods.
