Search papers, labs, and topics across Lattice.
ASR-TRA, a novel test-time adaptation framework, uses reinforcement learning to improve ASR robustness in noisy and accented speech environments. It introduces a learnable decoder prompt and temperature-controlled stochastic decoding to generate diverse transcription candidates, which are then scored by a reward model measuring audio-text semantic alignment. Experiments on LibriSpeech and L2 Arctic datasets demonstrate that ASR-TRA achieves higher accuracy and lower latency compared to existing test-time adaptation baselines.
Reinforcement learning with audio-text semantic rewards can overcome confirmation bias in test-time adaptation, leading to more robust ASR in noisy and accented speech environments.
Recently, Automatic Speech Recognition (ASR) systems (e.g., Whisper) have achieved remarkable accuracy improvements but remain highly sensitive to real-world unseen data (data with large distribution shifts), including noisy environments and diverse accents. To address this issue, test-time adaptation (TTA) has shown great potential in improving the model adaptability at inference time without ground-truth labels, and existing TTA methods often rely on pseudo-labeling or entropy minimization. However, by treating model confidence as a learning signal, these methods may reinforce high-confidence errors, leading to confirmation bias that undermines adaptation. To overcome these limitations, we present ASR-TRA, a novel Test-time Reinforcement Adaptation framework inspired by causal intervention. More precisely, our method introduces a learnable decoder prompt and utilizes temperature-controlled stochastic decoding to generate diverse transcription candidates. These are scored by a reward model that measures audio-text semantic alignment, and the resulting feedback is used to update both model and prompt parameters via reinforcement learning. Comprehensive experiments on LibriSpeech with synthetic noise and L2 Arctic accented English datasets demonstrate that our method achieves higher accuracy while maintaining lower latency than existing TTA baselines. Ablation studies further confirm the effectiveness of combining audio and language-based rewards, highlighting our method's enhanced stability and interpretability. Overall, our approach provides a practical and robust solution for deploying ASR systems in challenging real-world conditions.
