Search papers, labs, and topics across Lattice.
This paper introduces SpeechCombine, a novel approach to training instruction-following speech language models (SLMs) without the need for instruction tuning, which is traditionally more complex due to the unique characteristics of speech data. By leveraging a single round of speech pre-training on 30k hours of data and combining the weights of a text LLM with the differences from its instruction-tuned version, the authors demonstrate that this method effectively transfers capabilities from text to speech while maintaining performance. The results indicate that this approach could significantly reduce the reliance on extensive speech datasets, paving the way for more efficient SLM training.
Instruction-following speech models can be trained effectively without the cumbersome instruction tuning process, challenging the status quo in SLM development.
Instruction tuning for speech language models (SLMs) is substantially more challenging than for text-based large language models (LLMs), as it requires learning a new modality and a wide range of speech-specific instructions in addition to those supported by text LLMs. Existing SLM training approaches largely replicate the text LLM training paradigm by synthesizing large-scale speech pre-training and instruction-tuning datasets. However, this strategy is difficult to scale, since speech sequences are significantly longer than text sequences. In this paper, we propose SpeechCombine, an instruction-following speech language model trained without any instruction tuning, using only a single round of speech pre-training on 30k hours of data. Starting from a text LLM base model, we perform continuous pre-training on speech utterances to obtain a speech-adapted model, and then directly combine its weights with the weight difference between the instruction-tuned and base versions of the text LLM. Our results show that this simple combination strategy not only preserves the knowledge and capabilities of the original text LLM, but also effectively transfers them to the speech domain. These findings suggest a new direction for SLM training that avoids reliance on massive speech data.
