Search papers, labs, and topics across Lattice.
This paper introduces ELF-S2T, a novel audio-conditioned continuous-target generative model for speech-to-text (S2T) tasks that leverages the Embedded Language Flows (ELF) architecture. By integrating a frozen Whisper encoder and employing audio forcing during training, ELF-S2T effectively aligns recognition and translation processes in a continuous latent space, leading to competitive performance on benchmark datasets. The analysis reveals that both ASR and S2TT errors arise from similar underlying causes, suggesting a unified semantic mapping process that enhances our understanding of S2T systems.
Continuous-target modeling reveals a shared semantic mapping process in speech recognition and translation, challenging conventional discrete token generation approaches.
Speech-to-text (S2T) systems for recognition (ASR) and translation (S2TT) typically generate discrete text tokens. In contrast, continuous-target language modelling performs generation in a continuous space, yet its potential for S2T remains unexplored. To bridge this gap, we propose ELF-S2T, an audio-conditioned continuous-target generative model for S2T. Built upon the pre-trained Embedded Language Flows (ELF) backbone, ELF-S2T processes speech via a frozen Whisper encoder and a single linear projector, prepending the resulting audio condition to the noisy text latent for in-context, flow-matching denoising. To prevent the model from over-relying on its pre-trained text context, we introduce audio forcing during training, and further amplify the audio condition via classifier-free guidance at inference. Experiments on LibriSpeech and CoVoST2 show that ELF-S2T achieves competitive ASR and S2TT performance. Crucially, our error analysis reveals that, although ASR and S2TT errors look very different on the surface, both stem from the same underlying cause, a close distance confusion in the continuous latent space. This finding naturally aligns with the continuous representation generation paradigm, indicating a common semantic mapping process beneath recognition and translation. Our code and pretrained models are publicly available at https://github.com/Sslnon/ELF-S2T.
