Search papers, labs, and topics across Lattice.
The paper introduces SpokenTOD, a large-scale spoken task-oriented dialogue dataset with 52,390 dialogues and 1,034 hours of speech, augmented with realistic spoken user behaviors like cross-turn slots, barge-in, disfluency, and emotional prosody. Using this dataset, the authors develop SpokenUS, a spoken user simulator with a dedicated architecture for barge-in, designed to mimic human-like dialogue behavior. SpokenUS demonstrates comparable goal coverage to larger models and outperforms baselines in human evaluations, while also posing meaningful challenges to downstream agents.
A new spoken user simulator, SpokenUS, trained on a large-scale dataset, finally captures the messiness of real human conversation, including barge-ins and disfluencies, to better train dialogue agents.
Robust task-oriented spoken dialogue agents require exposure to the full diversity of how people interact through speech. Building spoken user simulators that address this requires large-scale spoken task-oriented dialogue (TOD) data encompassing spoken user behaviors, yet existing datasets are limited in scale and domain coverage, with no systematic pipeline for augmenting them. To address this, we introduce \textbf{SpokenTOD}, a spoken TOD dataset of 52,390 dialogues and 1,034 hours of speech augmented with four spoken user behaviors -- cross-turn slots, barge-in, disfluency, and emotional prosody -- across diverse speakers and domains. Building on SpokenTOD, we present \textbf{SpokenUS}, a spoken user simulator grounded in TOD with a dedicated architecture for barge-in. SpokenUS achieves comparable goal coverage to significantly larger models while substantially outperforming all baselines in Human MOS, disclosing slot values gradually across the dialogue as humans do rather than front-loading them. Further analysis confirms that SpokenUS's spoken behaviors pose meaningful challenges to downstream agents, making it a practical tool for training and evaluating more robust spoken dialogue systems.
