Search papers, labs, and topics across Lattice.
This study investigates the effect of translation-enhanced pre-training on speech encoders used in conjunction with Large Language Models (LLMs). By addressing the structural misalignment between language-specific representations of speech encoders and the unified space of LLMs, the authors propose that incorporating translation objectives can facilitate better alignment. The experimental results reveal that this approach significantly enhances cross-modal integration and improves performance on various downstream Speech LLM tasks.
Translation-enhanced pre-training not only aligns speech encoders with LLMs but also boosts performance across multiple speech tasks.
Connecting a pre-trained speech encoder to a Large Language Model (LLM) is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM. Unlike encoders based on automatic speech recognition, which often produce representations in separate language-specific spaces, LLMs operate within a unified language-agnostic space. A mechanism is required to align the encoder's language-specific representations with the LLM's shared space. We argue that speech translation provides a principled way to achieve this. Unlike monolingual transcription, translation requires the model to bridge different languages and learn language-agnostic representations. We experimentally evaluate the impact of incorporating translation objectives into speech encoder pre-training. Our results demonstrate that translation-enhanced pre-training improves cross-modal integration and leads to superior performance across downstream Speech LLM tasks.
