Search papers, labs, and topics across Lattice.
The paper introduces AVUR-LLM, a novel LLM-based audio-visual speech recognition (AVSR) system that uses sparse modality alignment and visual unit-guided refinement to improve robustness. AVUR-LLM enhances cross-modal alignment and complementary exchange between audio and visual features, reducing the LLM's computational load. Experiments on LRS3 show that AVUR-LLM achieves state-of-the-art results for AVSR, with a 37% relative improvement over the baseline system under additive-noise conditions at 0 dB SNR.
LLMs can achieve state-of-the-art audio-visual speech recognition by sparsely aligning modalities and refining with visual unit guidance, substantially boosting robustness in noisy environments.
Audio-Visual Speech Recognition (AVSR) integrates acoustic and visual information to enhance robustness in adverse acoustic conditions. Recent advances in Large Language Models (LLMs) have yielded competitive automatic speech recognition performance and shown effectiveness for AVSR. However, prior approaches project audio and visual features independently or apply shallow fusion, limiting cross-modal alignment and complementary exchange while increasing the LLM's computational load. To address this, we propose AVUR-LLM, an LLM-based Audio-Visual Speech Recognition via Sparse Modality Alignment and Visual Unit-Guided Refinement. Experiments on LRS3 demonstrate state-of-the-art results for AVSR. Under additive-noise conditions at 0 dB SNR, it achieves 37% relative improvement over the baseline system.
