Search papers, labs, and topics across Lattice.
The paper introduces Viseme-Guided AV-HuBERT (VisG AV-HuBERT), a multi-task fine-tuning framework for audio-visual speech recognition that incorporates auxiliary viseme classification to improve the model's reliance on visual articulatory features. By adding a lightweight viseme prediction sub-network to AV-HuBERT, the encoder is explicitly guided to preserve visual speech information. Experiments on LRS3 show VisG AV-HuBERT achieves significant WER improvements under noisy conditions, reducing errors by over 50% at -10dB SNR, demonstrating enhanced speech unit discrimination.
Explicitly modeling visemes in AV-HuBERT slashes word error rates by over 50% in noisy environments, proving that targeted visual guidance can dramatically improve speech recognition robustness.
Audio-Visual Speech Recognition (AVSR) systems nowadays integrate Large Language Model (LLM) decoders with transformer-based encoders, achieving state-of-the-art results. However, the relative contributions of improved language modelling versus enhanced audiovisual encoding remain unclear. We propose Viseme-Guided AV-HuBERT (VisG AV-HuBERT), a multi-task fine-tuning framework that incorporates auxiliary viseme classification to strengthen the model's reliance on visual articulatory features. By extending AV-HuBERT with a lightweight viseme prediction sub-network, this method explicitly guides the encoder to preserve visual speech information. Evaluated on LRS3, VisG AV-HuBERT achieves comparable or improved performance over the baseline AV-HuBERT, with notable gains under heavy noise conditions. WER reduces from 13.59% to 6.60% (51.4% relative improvement) at -10 dB Signal-to-Noise Ratio (SNR) for Speech noise. Deeper analysis reveals substantial reductions in substitution errors across noise types, demonstrating improved speech unit discrimination. Evaluation on LRS2 confirms generalization capability. Our results demonstrate that explicit viseme modelling enhances encoder representations, and provides a foundation for enhancing noise-robust AVSR through encoder-level improvements.
