Search papers, labs, and topics across Lattice.
This study investigates the interaction between speech and text modalities in interleaved speech language models (SLMs) by analyzing their latent space through a logit lens. The authors reveal that these models undergo an implicit transcription phase where spoken words are decoded into text tokens in intermediate layers, achieving a transcription accuracy of up to 77% for the data analyzed. This finding not only clarifies the internal mechanisms of SLMs but also suggests that the initialization from text language models enhances their spoken knowledge capabilities.
Interleaved speech-text models can implicitly transcribe spoken words into text with up to 77% accuracy, revealing a hidden layer of understanding in multimodal processing.
Speech language models (SLMs) have been extensively studied, with the common paradigm incorporating text data and pre-trained text LMs. A leading approach is speech-text interleaving in which models are trained over sequences containing both speech and text tokens, aiming to boost even speech-only capabilities. Yet the way these two modalities interact in the model latent space remains unclear. In this work, we analyze interleaved speech-text LMs from different model families and sizes through the scope of the logit lens to provide such insight. We reveal that these models go through an implicit transcription phase in which the text token of the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. The transcription of the word appears as one of the top candidate words for as much as 77\% of the data. Following this stage, the models proceed to predict the next word in the text space before transforming back to the speech domain. We finally analyze the role of interleaving data, and initializing from text LMs in eliciting this behavior, as well as seeing how this correlates with spoken knowledge abilities. Our analysis sheds light on the internal mechanisms underlying the relationship between speech and text modalities and could shape SLM optimization.
