Search papers, labs, and topics across Lattice.
This study conducts a mechanistic analysis of paralinguistic information in four audio language models, revealing that while these models strongly encode speaking style in the late encoder layers, this information is significantly degraded before reaching the output. By employing methods such as centered kernel alignment and a content prosody leakage metric, the research distinguishes between content-driven and acoustic-driven prediction behaviors across different architectures. The findings underscore a critical limitation in current audio language models, highlighting a disconnect between encoded information and its utilization in predictions.
Audio language models encode speaking style effectively but lose critical paralinguistic information before making predictions, revealing a significant gap in their capabilities.
Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.
