Search papers, labs, and topics across Lattice.
This paper analyzes LLM-based ASR systems through the lens of entropy allocation between the speech encoder and the LLM decoder, introducing three novel metrics to quantify this allocation. They identify inefficiencies in existing training paradigms and propose a multi-stage training strategy that includes a redesigned pretraining phase and an iterative asynchronous SFT stage. Experiments on Mandarin and English benchmarks demonstrate competitive ASR performance with a 2.3B parameter model and improved hallucination robustness.
LLM-based ASR models can achieve state-of-the-art performance and reduce hallucinations by strategically allocating entropy reduction between the speech encoder and LLM during training.
Integrating large language models (LLMs) into automatic speech recognition (ASR) has become a dominant paradigm. Although recent LLM-based ASR models have shown promising performance on public benchmarks, it remains challenging to balance recognition quality with latency and overhead, while hallucinations further limit real-world deployment. In this study, we revisit LLM-based ASR from an entropy allocation perspective and introduce three metrics to characterize how training paradigms allocate entropy reduction between the speech encoder and the LLM. To remedy entropy-allocation inefficiencies in prevailing approaches, we propose a principled multi-stage training strategy grounded in capability-boundary awareness, optimizing parameter efficiency and hallucination robustness. Specifically, we redesign the pretraining strategy to alleviate the speech-text modality gap, and further introduce an iterative asynchronous SFT stage between alignment and joint SFT to preserve functional decoupling and constrain encoder representation drift. Experiments on Mandarin and English benchmarks show that our method achieves competitive performance with state-of-the-art models using only 2.3B parameters, while also effectively mitigating hallucinations through our decoupling-oriented design.
