Search papers, labs, and topics across Lattice.
This paper introduces a unified post-training framework for speech foundation models that enables the generation of multiple utterance-level attribute representations from a single encoder. The approach extends existing methods like SAMU-XSLR and SONAR by allowing the model to learn arbitrary utterance-level attributes, rather than being limited to semantic representations. Experiments demonstrate the effectiveness of the framework by jointly learning semantic and speaker representations, achieving strong results on multilingual speech retrieval and speaker recognition.
Now a single speech foundation model can generate diverse utterance-level representations, like semantics and speaker identity, opening new possibilities for multimodal and multilingual applications.
Speech foundation models trained with self-supervised learning produce generic speech representations that support a wide range of speech processing tasks. When further adapted with supervised learning, these models can achieve strong performance on specific downstream tasks. Recent post-training approaches, such as SAMU-XSLR and SONAR, align speech representations with utterance-level semantic representations, enabling effective multimodal (speech-text) and multilingual applications. While speech foundation models typically learn contextual embeddings at the acoustic frame level, these methods learn representations at the utterance level. In this work, we extend this paradigm to arbitrary utterance-level attributes and propose a unified post-training framework that enables a single speech foundation model to generate multiple types of utterance-level representations. We demonstrate the effectiveness of this approach by jointly learning semantic and speaker representations and evaluating them on multilingual speech retrieval and speaker recognition tasks.
