Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates methods for integrating large language models (LLMs) into automatic speech recognition (ASR) systems, comparing tight integration of an acoustic model (AM) with the LLM to shallow fusion techniques. Through ablations on various factors like label units, fine-tuning strategies, and LLM sizes, the study identifies optimal configurations for both integration approaches. The research also explores joint recognition with a CTC model to reduce hallucinations and presents optimizations for this process, achieving state-of-the-art results on the HuggingFace ASR leaderboard.
Forget simply bolting on an LLM: this work reveals the surprisingly intricate dance between acoustic models and LLMs needed to unlock state-of-the-art speech recognition.
In this work, we study how to best utilize pre-trained LLMs for automatic speech recognition. Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion. For tight integration, we provide ablations on the effect of different label units, fine-tuning strategies, LLM sizes and pre-training data, attention interfaces, encoder downsampling, text prompts, and length normalization. Additionally, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations for this joint recognition. For shallow fusion, we investigate the effect of fine-tuning the LLM on the transcriptions using different label units, and we compare rescoring AM hypotheses to single-pass recognition with label-wise or delayed fusion of AM and LLM scores. We train on Librispeech and Loquacious and evaluate our models on the HuggingFace ASR leaderboard.