Search papers, labs, and topics across Lattice.
This paper introduces UBG-Net, an uncertainty-aware Bayesian gating network designed to enhance audio-visual speech recognition under challenging real-world conditions. By integrating a Modality Uncertainty-aware Bayesian Fusion mechanism, UBG-Net effectively models both aleatoric and epistemic uncertainties, leading to improved robustness in feature fusion. Experimental results on the AVCocktail and LRS2 datasets show that UBG-Net outperforms state-of-the-art baselines, particularly in noisy environments, confirming the effectiveness of its novel uncertainty modeling techniques.
UBG-Net outperforms existing models by effectively filtering noise and enhancing robustness in audio-visual speech recognition through advanced uncertainty modeling.
Audio-Visual speech recognition systems often degrade in real-world scenarios due to signal corruption and distribution shifts. To address this, we propose a unified uncertainty-modeling framework, namely the uncertainty-aware Bayesian gating network (UBG-Net). UBG-Net features a Modality Uncertainty-aware Bayesian Fusion (MUBF) mechanism that injects signal-level aleatoric uncertainty into a Bayesian network to model epistemic uncertainty, thereby ensuring robust fusion of pre-trained backbone features. For inference, we introduce Distribution Uncertainty-aware Hierarchical Voting (DUHV) to select transcripts from Monte Carlo samples, prioritizing frequency and using inference scores in case of a tie. Experiments on the AVCocktail and LRS2 datasets demonstrate the overall superiority of UBG-Net compared to SOTA baselines. Ablation studies confirm that MUBF and DUHV effectively filter noise, enhancing fusion and decoding robustness.
