Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel phoneme-first prediction approach for integrating Large Language Models (LLMs) with speech encoders to enhance speech recognition accuracy. By training the LLM to predict phonemes from speech features before generating transcripts, the method effectively reduces acoustic confusion and improves the model's understanding of pronunciation. Experimental results demonstrate significant improvements in transcription accuracy, particularly in low-resource settings, while also ensuring outputs are more acoustically faithful to the original speech.
Phoneme-first prediction boosts LLM-based speech recognition accuracy by reducing acoustic confusion and enhancing pronunciation understanding.
Recent research has explored integrating Large Language Models (LLMs) with speech encoders to create speech-augmented LLMs capable of contextualized speech recognition. The main challenge lies in aligning the semantic embeddings of LLMs with the acoustic representations of speech encoders. We propose a novel approach that teaches the LLM to first predict phonemes from the speech features before generating the final transcript. By integrating a phoneme prediction step directly into the LLM, the model develops a fine-grained knowledge of pronunciation, reducing acoustic confusion and improving transcription accuracy and explainability. Our method is cheap and simple, as phoneme targets can be automatically derived from existing transcripts. Through comprehensive experiments, we show that intermediate phoneme prediction can improve speech recognition, particularly in low-resource settings, and yields outputs that are acoustically more faithful to the speech.