Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ELF-S2T, an innovative audio-conditioned continuous-target generative model for speech-to-text (S2T) tasks that leverages a frozen Whisper encoder and a linear projector for improved performance. By employing audio forcing during training and classifier-free guidance at inference, ELF-S2T effectively bridges the gap between traditional discrete token generation and continuous representation modeling. Experimental results on LibriSpeech and CoVoST2 demonstrate that while ASR and S2TT errors manifest differently, they share a common root in the continuous latent space, suggesting a unified semantic mapping process for both tasks.
Continuous-target modeling reveals a shared semantic mapping for ASR and S2TT, challenging conventional views on their error sources.
Speech-to-text (S2T) systems for recognition (ASR) and translation (S2TT) typically generate discrete text tokens. In contrast, continuous-target language modelling performs generation in a continuous space, yet its potential for S2T remains unexplored. To bridge this gap, we propose ELF-S2T, an audio-conditioned continuous-target generative model for S2T. Built upon the pre-trained Embedded Language Flows (ELF) backbone, ELF-S2T processes speech via a frozen Whisper encoder and a single linear projector, prepending the resulting audio condition to the noisy text latent for in-context, flow-matching denoising. To prevent the model from over-relying on its pre-trained text context, we introduce audio forcing during training, and further amplify the audio condition via classifier-free guidance at inference. Experiments on LibriSpeech and CoVoST2 show that ELF-S2T achieves competitive ASR and S2TT performance. Crucially, our error analysis reveals that, although ASR and S2TT errors look very different on the surface, both stem from the same underlying cause, a close distance confusion in the continuous latent space. This finding naturally aligns with the continuous representation generation paradigm, indicating a common semantic mapping process beneath recognition and translation. Our code and pretrained models are publicly available at https://github.com/Sslnon/ELF-S2T.