Search papers, labs, and topics across Lattice.
This paper investigates the impact of layer pruning on the Whisper encoder within a SLAM-ASR system, focusing on the trade-off between model size and ASR performance. They find that pruning two encoder layers leads to only minor WER degradation (2-4%) and that LoRA fine-tuning can not only recover this loss but also surpass the unpruned baseline. Error analysis reveals that LoRA primarily compensates through the language model's linguistic priors, with varying effectiveness depending on the language resource level.
You can slash 7-14% of parameters from your SLAM-ASR system by pruning the Whisper encoder and using LoRA, even outperforming the original model in some cases.
Automatic speech recognition (ASR) has advanced rapidly in recent years, driven by large-scale pretrained models and end-to-end architectures such as SLAM-ASR. A key component of SLAM-ASR systems is the Whisper speech encoder, which provides robust acoustic representations. While model pruning has been explored for the full Whisper encoder-decoder architecture, its impact within the SLAM-ASR setting remains under-investigated. In this work, we analyze the effects of layer pruning in the Whisper encoder when used as the acoustic backbone of SLAM-ASR. We further examine the extent to which LoRA-based fine-tuning can recover performance degradation caused by pruning. Experiments conducted across three Whisper variants (Small, Medium, Large-v2), three languages representing distinct resource levels (Danish, Dutch, English), and over 200 training runs demonstrate that pruning two encoder layers causes only 2-4% WER degradation, and that combining this pruning with LoRA adaptation consistently outperforms the unpruned baseline while reducing total parameters by 7-14%. Moreover, our error analysis reveals that LoRA primarily compensates through the language model's linguistic priors, reducing total word errors by 11-21% for Dutch and English, with substitutions and deletions showing the largest reductions. However, for low-resource Danish, the reduction is smaller (4-7%), and LoRA introduces increased insertion errors, indicating that compensation effectiveness depends on the LLM's pre-existing language proficiency and available training data.
