Search papers, labs, and topics across Lattice.
This paper introduces a reliability-aware scaling method for contrastive decoding (CD) in audio-visual speech recognition (AVSR) systems, addressing the trade-off between noise robustness and prediction accuracy. By adaptively modulating the contrastive influence based on attention dynamics and inter-model predictive divergence, the authors enhance the system's performance across varying noise levels. Experimental results on the LRS3 dataset demonstrate significant improvements in recognition accuracy under both clean and low-SNR conditions, highlighting the effectiveness of the proposed approach.
Adaptive contrastive decoding boosts audio-visual speech recognition performance, striking the right balance between noise resilience and prediction fidelity.
Large language model (LLM)-based audio-visual speech recognition (AVSR) systems are robust under noise. Contrastive decoding (CD), originally introduced to stabilize LLM generation by contrasting a weaker model against a stronger one at inference time, adjusts predictions without additional training. In this work, we apply CD to AVSR by contrasting audio-only conditioning with full audio-visual conditioning within the same underlying model. However, using a fixed contrastive strength introduces a trade-off across noise levels: stronger intervention helps under severe noise but may over-correct reliable predictions in clean conditions. We propose reliability-aware scaling of CD for AVSR. Instead of using a fixed strength, we adaptively modulate the contrastive influence at each token based on reliability signals derived from attention dynamics and inter-model predictive divergence. Experiments on LRS3 show consistent improvements across clean and low-SNR conditions.
