Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates how factual knowledge is encoded and recalled in multimodal Speech Language Models (SLMs) by comparing text-to-text and speech-to-text performance. Using Causal Mediation Analysis on SpiritLM, the authors find that the mechanisms for factual recall do not fully transfer from the text modality to the speech modality. Specifically, discrepancies in mediation effects suggest different internal representations and retrieval processes for factual associations depending on the input modality.
Speech-enabled AI may require fundamentally different knowledge encoding strategies than text-based models, as factual recall mechanisms don't fully transfer between modalities.
In recent years, several Speech Language Models (SLMs) that represent speech and written text jointly have been presented. The question then emerges about how model-internal mechanisms are similar and different when operating in the two modalities. We focus on how these systems encode, store, and retrieve factual knowledge, which has previously been investigated for text-only models. To investigate mechanisms behind the storage and recall of factual association in SLMs, we leverage Causal Mediation Analysis, a technique previously applied to text-based models. Initial results using SpiritLM, a multimodal model integrating discrete speech tokens reveal discrepancies between text-to-text and speech-to-text results, suggesting that the emergent mechanisms for factual recall are only partially carried over from the text to the speech modality. These results advance our understanding of how internal mechanisms encode factual associations in SLMs while contributing insights for improving speech-enabled AI systems.