Search papers, labs, and topics across Lattice.
This paper introduces Self-Aware Speech Language Models (SA-SLM) to address the semantic understanding-acoustic realization gap in speech generation. SA-SLM incorporates Intent-Aware Bridging using a Variational Information Bottleneck to ensure stable, utterance-level expressive intent, and Realization-Aware Alignment, where the model critiques its own acoustic output to align it with the intended expression. Experiments on 800 hours of expressive speech data show that a 3B parameter SA-SLM outperforms open-source baselines and approaches the performance of GPT-4o-Audio in expressiveness.
Forget flat, lifeless speech: this model uses self-critique to generate expressive speech rivaling GPT-4o-Audio, even with significantly less training data.
Speech Language Models (SLMs) exhibit strong semantic understanding, yet their generated speech often sounds flat and fails to convey expressive intent, undermining user engagement. We term this mismatch the semantic understanding-acoustic realization gap. We attribute this gap to two key deficiencies: (1) intent transmission failure, where SLMs fail to provide the stable utterance-level intent needed for expressive delivery; and (2) realization-unaware training, where no feedback signal verifies whether acoustic outputs faithfully reflect intended expression. To address these issues, we propose SA-SLM (Self-Aware Speech Language Model), built on the principle that the model should be aware of what it thinks during generation and how it speaks during training. SA-SLM addresses this gap through two core contributions: (1) Intent-Aware Bridging, which uses a Variational Information Bottleneck (VIB) objective to translate the model's internal semantics into temporally smooth expressive intent, making speech generation aware of what the model intends to express; and (2) Realization-Aware Alignment, which repurposes the model as its own critic to verify and align acoustic realization with intended expressive intent via rubric-based feedback. Trained on only 800 hours of expressive speech data, our 3B parameter SA-SLM surpasses all open-source baselines and comes within 0.08 points of GPT-4o-Audio in overall expressiveness on the EchoMind benchmark.
