Search papers, labs, and topics across Lattice.
This study conducts a layer-wise analysis of the Whisper ASR model to understand how multilingual medical adaptation affects its internal representations. By comparing various fine-tuning strategies, the authors find that while fine-tuning significantly enhances MedASR performance, the optimal model varies based on the adaptation context, with Whisper-Medium achieving the lowest English WER and German-only Whisper-Large-v3 excelling in German WER. The analysis reveals that English medical fine-tuning leads to a pronounced encoder shift, while multilingual continuation maintains the adapted representation space, indicating nuanced interactions between domain and language information across layers.
Fine-tuning Whisper models for multilingual medical ASR reveals that the best performance hinges on the adaptation strategy, with surprising shifts in internal representations based on language context.
Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong generalisation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN->EN+DE continuation, and direct EN+DE fine-tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper-Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two-stage Whisper-Small trajectory shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves.
