Search papers, labs, and topics across Lattice.
This paper introduces Gumbel-BEARD, a novel framework for automating layer selection in the Whisper speech model to enhance performance in low-resource domains. By employing an end-to-end trainable Gumbel-Softmax selector, the method enables self-supervised adaptation that dynamically aligns with target acoustic characteristics, achieving competitive results with minimal labeled data. Experiments reveal that Gumbel-BEARD achieves state-of-the-art word error rates (WERs) of 8.21% and 11.06% on two different datasets, demonstrating its efficiency and robustness across various dialectal shifts.
Achieving state-of-the-art performance with just 10 hours of labeled data, Gumbel-BEARD revolutionizes self-supervised adaptation for speech models in low-resource settings.
Speech foundation models often struggle in low-resource domains due to domain mismatch and data scarcity. We propose Gumbel-BEARD, a domain adaptation framework that automates Whisper encoder layer selection via an end-to-end trainable hard Gumbel-Softmax selector. It enables self-supervised adaptation with a BEST-RQ objective that dynamically adapts to target acoustic characteristics without manual tuning. Experiments on the MyST child speech corpus demonstrate efficiency and scalability: with 10 h of labeled data for fine-tuning, our method matches a fully supervised baseline trained on the complete 133 h labeled set. We establish new state-of-the-art word error rates (WERs) of 8.21% using Whisper-medium on MyST and 11.06% using Whisper-small on the OGI Spontaneous dataset. Evaluation on CORAAL further confirms robustness to adult dialectal domain shifts, with up to 6% relative WER reduction, highlighting the generalizability of our approach to diverse low-resource conditions.
