Search papers, labs, and topics across Lattice.
This paper addresses the problem of AI assistants interrupting in multi-party dialogues by formulating context-aware turn-taking, where the assistant decides whether to speak or stay silent based on the conversation context. They introduce a benchmark dataset of 120K labeled conversations and demonstrate that large language models perform poorly in zero-shot settings on this task. Supervised fine-tuning with reasoning traces significantly improves performance, suggesting that context-aware turn-taking requires explicit training.
LLMs can't tell when to shut up in multi-party conversations, but fine-tuning with reasoning traces can teach them some manners.
Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capability; it must be explicitly trained.
