Search papers, labs, and topics across Lattice.
This paper introduces X2-Turn, a frame-synchronous dual-head modeling approach that simultaneously predicts turn states and automatic speech recognition (ASR) tokens in real-time. By leveraging a shared streaming representation and a parallel architecture, X2-Turn addresses the limitations of prior modular methods that struggle with continuous turn state estimation and responsiveness. Evaluated on bilingual Chinese-English datasets, the method achieves high accuracy in turn-taking detection while maintaining low latency, outperforming existing approaches.
Real-time turn-taking detection can be achieved with unprecedented accuracy and low latency using a novel dual-head modeling approach.
Accurate and responsive turn-taking is essential for spoken dialogue systems, which must distinguish in real time between user interruptions, backchannels that should be ignored, and the completion of an utterance. Prior modular approaches typically optimize turn state prediction at the utterance or fixed-chunk level, creating a mismatch with the continuous turn state estimate, and often depend on an auxiliary ASR model, which limits responsiveness and increases overall system complexity. Therefore, we present X2-Turn, a frame-synchronous turn state prediction method via delayed-stream modeling. Specifically, building on the pretrained Voxtral Realtime model, we introduce a frame-synchronous turn state head that operates in parallel with the ASR head on shared streaming representations, jointly predicting ASR tokens and fine-grained turn states at the frame level. We evaluate our method on the bilingual Chinese-English Easy-Turn test sets, and the results demonstrate its effectiveness in achieving accurate turn-taking detection while maintaining low latency.
