Search papers, labs, and topics across Lattice.
WavSLM is introduced as a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook. It optimizes an autoregressive next-chunk prediction objective, modeling semantic and acoustic information jointly within a single token stream without text supervision. WavSLM achieves competitive performance on consistency benchmarks and speech generation with fewer parameters, less training data, and streaming inference.
Ditch the text: WavSLM shows you can train a competitive speech language model using only distilled WavLM representations, unlocking a simpler, single-stream generative pretraining paradigm for speech.
Large language models show that simple autoregressive training can yield scalable and coherent generation, but extending this paradigm to speech remains challenging due to the entanglement of semantic and acoustic information. Most existing speech language models rely on text supervision, hierarchical token streams, or complex hybrid architectures, departing from the single-stream generative pretraining paradigm that has proven effective in text. In this work, we introduce WavSLM, a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook and optimizing an autoregressive next-chunk prediction objective. WavSLM jointly models semantic and acoustic information within a single token stream without text supervision or text pretraining. Despite its simplicity, it achieves competitive performance on consistency benchmarks and speech generation while using fewer parameters, less training data, and supporting streaming inference. Demo samples are available at https://lucadellalib.github.io/wavslm-web/.
Mila