Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a training method for speech-LLMs that leverages broad descriptions from video metadata to enhance contextual reasoning in automatic speech recognition (ASR). By generating reasoning-augmented speech data, the authors fine-tune the model to perform chain-of-thought reasoning, which significantly reduces errors, particularly with rare words and named entities. The results demonstrate a promising advance in contextual understanding for ASR systems, paving the way for more robust speech recognition capabilities.
Contextual reasoning in speech recognition can be dramatically improved by grounding models in rich video metadata, leading to significant error reductions in rare terms and named entities.
Speech recognition often fails on rare, domain-specific terms and context-related named entities. Existing contextualization techniques typically bias decoding with keywords or phrase lists, which does not scale well or exploit deeper knowledge. We propose a training method that teaches a speech-LLM to use broad descriptions (e.g. from videos) as weak semantic priors to perform contextual reasoning grounded in the audio. We build 400 hours of reasoning-augmented speech data by pairing erroneous hypotheses with video metadata and LLM-generated reasoning explanations that justify context-driven corrections. We finetune the speech-LLM to perform chain-of-thought reasoning: generate an initial transcript, then reason over the context, and finally return a corrected transcript. On held-out YouTube-derived test sets, our approach reduces errors, with specific improvements on rare words and named entities, and lays groundwork for deeper contextual reasoning in speech recognition.