Search papers, labs, and topics across Lattice.
This paper explores the adaptation of LLM-based automatic speech recognition (ASR) systems in regulated domains, specifically banking, where privacy constraints limit the use of real speech data. By employing Group Relative Policy Optimization (GRPO) instead of traditional supervised fine-tuning (SFT), the authors achieve a significant reduction in word error rate (WER) by 40% relative to SFT, demonstrating that GRPO effectively leverages synthetic speech to improve performance. The findings highlight that GRPO enhances model behavior, particularly in reducing insertion errors and improving speech-to-text alignment, while maintaining early-layer representations intact.
Reinforcement learning outperforms supervised fine-tuning in adapting ASR systems to synthetic speech, achieving a 40% reduction in word error rates.
LLM-based ASR adapted to regulated domains such as banking is bottlenecked by privacy: real speech is costly and legally constrained to collect, making synthetic text-to-speech (TTS) an attractive substitute. Yet synthetic speech stays acoustically mismatched with real recordings, and work on this gap has stayed within supervised fine-tuning (SFT). We instead turn to reinforcement learning, and show that Group Relative Policy Optimization (GRPO) extracts far more from the same synthetic speech than SFT. Synthetic-only adaptation of the model with GRPO, a critic-free method rewarding low-WER hypotheses, reduces WER by 40\% relative to SFT (36.71\%$\to$22.09\%), and an SFT-then-GRPO combination pushes this further to 45\%. We trace the gain to behavior rather than representation: GRPO reduces insertion errors by improving stopping calibration and speech-to-text alignment by better anchoring attention to audio, leaving early-layer representations intact. When synthetic speech is the main resource, reinforcement learning should be preferred over supervised fine-tuning.
