Search papers, labs, and topics across Lattice.
This paper investigates two approaches for leveraging pre-trained LLMs in automatic speech recognition: tight integration of an acoustic model (AM) with the LLM ("speech LLM") and shallow fusion of AM and LLM. The authors perform extensive ablations on various aspects of tight integration, including label units, fine-tuning strategies, LLM sizes, and attention interfaces, and explore joint recognition with CTC to reduce hallucinations. Through experiments on Librispeech and Loquacious, the study provides insights into the effectiveness of different integration and combination strategies, benchmarked on the HuggingFace ASR leaderboard.
Fine-tuning LLMs directly on speech can lead to powerful ASR systems, but requires careful tuning of label units, architectures, and training strategies to avoid hallucinations.
In this work, we study how to best utilize pre-trained LLMs for automatic speech recognition. Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion. For tight integration, we provide ablations on the effect of different label units, fine-tuning strategies, LLM sizes and pre-training data, attention interfaces, encoder downsampling, text prompts, and length normalization. Additionally, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations for this joint recognition. For shallow fusion, we investigate the effect of fine-tuning the LLM on the transcriptions using different label units, and we compare rescoring AM hypotheses to single-pass recognition with label-wise or delayed fusion of AM and LLM scores. We train on Librispeech and Loquacious and evaluate our models on the HuggingFace ASR leaderboard.
