Search papers, labs, and topics across Lattice.
This study explores the use of aligned audio-image pairs to enhance Automatic Speech Recognition (ASR) systems for low-resource languages by introducing a representation alignment stage between large-scale pretraining and supervised fine-tuning. By leveraging the Vaani dataset, which provides paired audio and image data, the authors adapt pretrained audio encoders without the need for costly transcription data. The results indicate that models undergoing this alignment process significantly outperform those that are directly fine-tuned, showcasing a promising transcription-free strategy for ASR improvement in under-resourced languages.
Aligning audio with image representations can drastically boost ASR performance in low-resource languages without the need for expensive transcriptions.
Thousands of languages are spoken worldwide, yet many remain under-resourced for Automatic Speech Recognition (ASR) due to the limited availability of high-quality transcribed speech data. Collecting accurate transcriptions is often costly and labor-intensive, particularly for low-resource languages. In this work, we investigate the use of aligned audio-image pairs to adapt pretrained audio encoders without requiring transcription data before supervised fine-tuning. Our proposed representation alignment stage is introduced between large-scale pretraining and supervised ASR fine-tuning. Specifically, image representations extracted from pretrained vision encoders are aligned with audio representations to further adapt a pretrained audio encoder. For this alignment process, we utilize the Vaani dataset, in which images serve as prompts for speech collection, naturally providing paired audio-image data. We evaluate the proposed approach using multiple vision encoders and a pretrained FastConformer audio encoder. Experimental results demonstrate that models fine-tuned after representation alignment consistently achieve improved ASR performance compared to direct fine-tuning. These findings highlight the potential of audio-image representation alignment as an effective transcription-free adaptation strategy for enhancing ASR systems in low-resource language settings.
