Search papers, labs, and topics across Lattice.
This paper introduces Joint Speech-Text Interleaved Pretraining (JSTIP), a novel approach to enhance automatic speech recognition (ASR) by interleaving speech and text sequences during training. The authors demonstrate that as the amount of supervised ASR data increases, the benefits of large language model (LLM) priors diminish, highlighting the need for more effective integration strategies. Experiments reveal that JSTIP significantly improves entity recognition accuracy and performs competitively with existing ASR and Speech-LLM systems, particularly in medical contexts, while also facilitating zero-shot speech question answering.
Interleaving speech and text during ASR training boosts entity recognition accuracy and narrows the gap between modalities, challenging traditional training paradigms.
Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recognition (ASR) remain unclear. We observe that as supervised ASR training data increases, the contribution of LLM priors becomes less evident, and simple speech-text joint training under-utilizes textual knowledge. We therefore propose Joint Speech-Text Interleaved Pretraining (JSTIP), an ASR-oriented pretraining strategy that constructs word-level and segment-level interleaved speech-text sequences within aligned pairs for speech-LLM architectures that accept continuous inputs. Experiments on 38k hours of ASR data show consistent entity accuracy improvement compared to ASR-only and joint speech-text training baselines. JSTIP achieves on-par entity recognition performance using domain transcription text compared to synthetic speech-text pairs, simplifying domain adaptation. Benefiting from textual pretraining and domain text data, JSTIP is competitive with open-source ASR and Speech-LLM systems in medical entity recognition. The zero-shot speech question answering behaviors further suggest that interleaving reduces the speech-text modality gap and preserves the LLM generative prior, which is likely the reason for the entity improvements on the ASR task.
Microsoft Research