Search papers, labs, and topics across Lattice.
This paper investigates the internal predictive behavior of full-duplex spoken language models (FD-SLMs) and identifies a phenomenon termed "state inertia," where the models exhibit a delayed transition between generative and perceptive states during user interruptions. By introducing the Zero-Buffer Benchmark (ZBB), the authors quantify the impact of this inertia on comprehension during abrupt conversational changes. They propose a novel intervention called activation steering, which enhances the models' ability to handle interruptions, resulting in significant improvements in response correctness and initial-word occurrence rates without requiring fine-tuning.
State inertia in full-duplex spoken language models can lead to missed user input, but activation steering effectively mitigates this issue, boosting comprehension rates significantly.
Full-duplex spoken language models (FD-SLMs) enable seamless speech interaction by allowing models to listen and speak simultaneously, yet the internal mechanism by which they coordinate listening and speaking remains underexplored. We analyze the predictive behavior encoded in FD-SLM hidden representations and find that they exhibit stream-specific predictive patterns: during listening, they preferentially predict the incoming user stream, whereas during speaking, they preferentially predict the model output stream. Building on this observation, we show that FD-SLMs dynamically modulate their internal predictive focus between two states: a generative state aligned with model output generation and a perceptive state aligned with incoming user input. However, this modulation can lag behind abrupt changes in conversational context. During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it to miss the beginning of the incoming input. We term this delayed internal transition state inertia. To quantify its downstream impact, we introduce the Zero-Buffer Benchmark (ZBB), a diagnostic benchmark for evaluating immediate interruption comprehension when user speech begins abruptly. We evaluate this setting using response correctness and initial-word occurrence rate (IWOR). Finally, we mitigate state inertia through activation steering with a perception vector, a training-free intervention with little additional computational overhead. Across multiple state-of-the-art FD-SLMs, activation steering substantially improves interruption handling; for example, on PersonaPlex, it improves correctness from 28% to 45% and IWOR from 40% to 72% without any fine-tuning.
MIT CSAIL