Search papers, labs, and topics across Lattice.
Full-duplex speech models frequently suffer from "anchor interruptions"鈥攊nconsistencies caused by latency mismatches between internal text generation, TTS synthesis, and the audio actually played to the user. To ground interruption recovery in what the user genuinely heard, the authors introduce Self-Listening, a multi-stream framework that feeds the model's realized acoustic playback back into its input context alongside user audio. Evaluated on AnchorSpeech, a new benchmark measuring state tracking across conversational interruptions, self-listening substantially outperforms standard full-duplex baselines in maintaining conversational continuity.
Interrupted voice agents often hallucinate what they have already said because text generation outpaces audio playback鈥攁 failure mode solved by forcing the model to listen to its own real-time voice output.
Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the model's realized speech as anchor interruption. To address this problem, we propose Self-Listening, a full-duplex modeling approach that interleaves user speech, model text, and the model's played speech. By feeding the realized speech output back to the model as an input stream, self-listening grounds interruption recovery in what the user has actually heard. We further introduce AnchorSpeech, a collection with homogeneous training and test splits for tracking which items of structured ordered responses have actually been spoken. AnchorSpeech-test evaluates whether a model can respond consistently with the last completed item before an interruption. Experiments show that, compared with full-duplex baselines, models equipped with self-listening mechanism achieve better anchoring performance.
