Search papers, labs, and topics across Lattice.
This paper addresses the issue of language adherence in multimodal LLMs used for Automatic Speech Recognition (ASR), where misidentification of output languages can lead to poor transcription quality. The authors propose a soft prompting approach that allows for flexibility in language use while introducing a novel metric to quantify language adherence violations. Through a comparative analysis of three mitigation strategies鈥攝ero-shot prompting, supervised fine-tuning, and Chain-of-Thought reasoning鈥攖he study demonstrates significant improvements in language adherence without sacrificing ASR performance across multiple languages.
Language adherence in ASR can be significantly improved using soft prompting techniques, leading to better transcription quality in multilingual contexts.
While Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use, models often misidentify the output language, compromising transcription fidelity and downstream application quality. To preserve flexibility and code-switching capabilities, we propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output. We formally define this challenge as a lack of language adherence, introduce a novel metric to quantify violations, and evaluate three mitigation strategies: (1) zero-shot prompting for robust guidance under uncertainty, (2) supervised fine-tuning (SFT) to improve prompt adherence, and (3) Chain-of-Thought (CoT) reasoning to enforce adherence during decoding. We present a comparative analysis of these methods across multiple languages, evaluating effectiveness in reducing the language violation while maintaining overall ASR performance. Finally, we discuss trade-offs to guide strategy selection under various compute constraints.
DeepMind