Search papers, labs, and topics across Lattice.
This paper introduces VIB-AVSR, a novel approach that integrates Variational Information Bottleneck layers into LLM-based audio-visual speech recognition systems to enhance robustness against noisy audio inputs. By strategically placing these layers within the LLM backbone, the model effectively regularizes representations, mitigating performance degradation in adverse acoustic conditions. The results demonstrate significant improvements in recognition accuracy across various signal-to-noise ratios and noise types, highlighting the method's efficacy without necessitating architectural changes or extra training data.
Noise doesn't stand a chance against VIB-AVSR, which boosts LLM-based audio-visual speech recognition performance by integrating Variational Information Bottleneck layers for enhanced robustness.
Audio-Visual Speech Recognition takes two input modalities, acoustic and visual streams, where visual information from lip movements aids recognition when audio is noisy. Recently, LLM-based AVSR models have emerged as a promising paradigm by connecting pre-trained audio-visual encoders to an LLM, achieving strong results in clean conditions. However, these models are predominantly optimized for clean acoustic conditions, with limited attention to making the LLM backbone robust to noise. No explicit mechanism is employed to produce stable representations under corrupted audio, leading to performance degradation in noisy environments. To address this, we propose VIB-AVSR, which integrates Variational Information Bottleneck layers at targeted positions within the LLM backbone to regularize representations. VIB-AVSR reduces degradation under noisy conditions across multiple SNR levels and noise types, without requiring architectural modifications or additional training data.
