Search papers, labs, and topics across Lattice.
This paper introduces Endpoint Anticipation, a proactive approach to forecasting end-of-turn signals in spoken dialogue systems, which addresses the latency issues associated with traditional reactive turn-completion detection. By anticipating endpoints up to 2.56 seconds in advance, the proposed model allows for speculative execution of large language models (LLMs) and text-to-speech (TTS) pipelines, significantly improving real-time interaction. Evaluation shows that this method achieves a 505 ms average latency reduction while increasing speculative computation by 28.4%, outperforming existing VAP-based baselines in both conversational and task-oriented contexts.
Anticipating dialogue endpoints up to 2.56 seconds ahead can slash latency by over half while enhancing computational efficiency in real-time speech interactions.
While low-latency interaction is critical for spoken dialogue, cascaded architectures are often bottlenecked by reactive turn-completion detection. We propose Endpoint Anticipation, shifting from reactive detection to proactive forecasting of end-of-turn signals. Our speech-based model anticipates endpoints upto 2.56 seconds in advance, enabling speculative execution of LLM and TTS pipelines on partial context. We introduce metrics to quantify the trade-off between realized latency reduction and computational redundancy. Evaluation across conversational and task-oriented datasets shows our model consistently outperforms competitive VAP-based baselines. Integration with the Unmute framework demonstrates a 505 ms average latency reduction with a 28.4% increase in speculative computation, effectively masking sequential bottlenecks to enable complex reasoning in real-time speech-to-speech interaction.
CMU ML