Search papers, labs, and topics across Lattice.
This paper introduces a decoupled data approach that enhances turn-taking in dialogue systems by learning from real Human-Human spoken dialogues while maintaining semantic integrity through Human-Agent text dialogues. By employing a rule-based event-guided data transformation method, the authors effectively serialize spoken dialogues into finite-state machine (FSM) tapes, allowing for scalable supervision without reliance on LLM-generated annotations. The proposed Source-Aware Calibrated (SAC) Loss significantly improves turn-taking proficiency and preserves the foundational semantic capabilities of large language models, demonstrating a notable advancement in full-duplex dialogue systems.
Real Human-Human dialogue data can transform turn-taking in dialogue systems, achieving better proficiency without sacrificing semantic quality.
The Neural Finite State Machine (NFSM) framework offers a pragmatic path to full-duplex dialogue by serializing turn-taking control and response generation onto a single causal tape under the standard next-token prediction objective, thereby preserving semantic prowess at a low fine-tuning cost. However, its reliance on synthetic text data fundamentally limits turn-taking naturalness, as Large Language Models (LLMs) cannot faithfully simulate the fine-grained acoustic temporal dynamics of real human dialogues. In this work, we propose a decoupled data approach that learns turn-taking from real Human-Human (HH) spoken dialogues while shaping semantic behavior through configurable Human-Agent (HA) text dialogues. To operationalize this approach, we introduce a rule-based event-guided data transformation method that serializes HH spoken dialogues into FSM tapes by classifying turn-taking events and applying deterministic mapping rules, enabling scalable supervision without LLM-generated annotations. We further propose a Source-Aware Calibrated (SAC) Loss that jointly calibrates the long-tailed distribution of state transition tokens and channels each data source toward the capability it best supervises. Experiments show that our approach substantially improves turn-taking proficiency while recovering the foundation LLM's semantic capability. Our code and model are available at https://github.com/Liyht/def-fsm.
