Search papers, labs, and topics across Lattice.
This paper introduces MELD, a discrete latent variable model for speech language modeling that jointly optimizes the encoder and the speech language model directly on mel spectrograms. By jointly training the encoder and decoder, MELD overcomes limitations of separately trained encoders and improves performance on zero-shot TTS and STT tasks. The model also mitigates common issues in autoregressive mel-spectrogram modeling, such as prolonged silence and word omissions.
Jointly training a speech encoder and language model on mel-spectrograms not only boosts zero-shot speech translation, but also fixes annoying speech synthesis quirks like endless silences.
Recent speech language models rely on encoders that are optimized separately from autoregressive models. Since these encoders are unaware of the downstream objectives, the extracted representations may not be optimal for downstream tasks. To address this limitation, we introduce a discrete latent variable model on mel spectrograms that jointly optimizes the encoder and the speech language model. Joint optimization not only brings improvements over codec-based and other mel-spectrogram-based baselines on zero-shot Text-to-Speech (TTS) and Speech-to-Text (STT) tasks, but also effectively alleviates common issues in autoregressive mel-spectrogram modeling, such as prolonged silence generation and word omissions.
Meta AI