Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Discourse-Aware Dual-Track Streaming Response (DDTSR), a novel framework for low-latency spoken dialogue systems that overlaps ASR, LLM inference, and TTS processes. DDTSR employs a connective-guided small-large model synergy to generate discourse connectives in parallel with knowledge-intensive reasoning by a large model. Experiments on two spoken dialogue benchmarks demonstrate a 19%-51% reduction in response latency while maintaining discourse quality, showcasing the framework's compatibility with diverse LLMs and robustness across varying utterance lengths.
Slash spoken dialogue system latency by up to 51% with a new architecture that lets the system "listen-while-thinking" and "speak-while-thinking."
Achieving human-like responsiveness is a critical yet challenging goal for cascaded spoken dialogue systems. Conventional ASR-LLM-TTS pipelines follow a strictly sequential paradigm, requiring complete transcription and full reasoning before speech synthesis can begin, which results in high response latency. We propose the Discourse-Aware Dual-Track Streaming Response (DDTSR) framework, a low-latency architecture that enables listen-while-thinking and speak-while-thinking. DDTSR is built upon three key mechanisms: (1) connective-guided small-large model synergy, where an auxiliary small model generates minimal-committal discourse connectives while a large model performs knowledge-intensive reasoning in parallel; (2) streaming-based cross-modal collaboration, which dynamically overlaps ASR, LLM inference, and TTS to advance the earliest speakable moment; and (3) curriculum-learning-based discourse continuity enhancement, which maintains coherence and logical consistency between early responses and subsequent reasoning outputs. Experiments on two spoken dialogue benchmarks demonstrate that DDTSR reduces response latency by 19%-51% while preserving discourse quality. Further analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.