Search papers, labs, and topics across Lattice.
This paper investigates the alignment between speech and text representations in Spoken Language Models (SLMs) and identifies significant structural gaps that hinder their instruction-following capabilities and generalization across tasks. By proposing a framework that decouples length mismatch from semantic alignment, the authors enhance the correspondence between speech and text, leading to improved performance on various benchmarks. The results highlight the necessity of addressing these structural differences in SLM training to achieve competitive outcomes against traditional text-based models.
Despite strong downstream performance, SLMs struggle with instruction-following due to weak alignment between speech and text representations, revealing a critical gap in their training methodology.
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.
