Search papers, labs, and topics across Lattice.
This paper introduces ModeratorLM, a role-playing voice agent designed to enhance turn-taking in multi-party conversations by conditioning its behavior on explicitly assigned roles. By leveraging a speech large language model in a chunk-wise streaming manner and incorporating chain-of-thought reasoning, the system significantly improves turn-taking precision and recall. Experiments demonstrate that ModeratorLM outperforms non-role-conditioned baselines, achieving over 40% improvement in precision and more than 70% in recall, while minimizing false-positive interruptions.
Role-conditioned turn-taking can boost conversational precision by over 40%, transforming how voice agents manage multi-party interactions.
Turn-taking in multi-party spoken conversations remains a fundamental challenge for voice-based agents, particularly under dynamic floor competition and varying user expectations. We propose ModeratorLM, a role-playing voice agent that conditions turn-taking behavior on an explicitly assigned role in multi-party settings. The system is built on a speech large language model operating in chunk-wise streaming manner. We further introduce a reasoning-augmented variant that incorporates chain-of-thought reasoning over conversational context and the assigned role. We construct RolePlayConv, a large-scale synthetic dataset of spoken multi-party conversations with diverse assistant roles. Experiments on real-world meeting data and RolePlayConv show improved turn-taking precision by over 40% and recall by more than 70%, while substantially reducing false-positive interruptions compared to non-role-conditioned baselines.
Amazon Science