Search papers, labs, and topics across Lattice.
This paper introduces Cocktail-Talker, a framework for modeling multi-speaker dialog in noisy social environments, addressing the challenges posed by multiple speakers and background noise. The system employs three action tokens—<|respond|>, <|listen|>, and <|ignore|>—to guide the assistant's interaction decisions, allowing it to determine when to respond or remain silent. Trained through a combination of supervised fine-tuning and reinforcement learning, Cocktail-Talker demonstrates improved performance in generating contextually appropriate responses in complex conversational settings.
Cocktail-Talker enables spoken dialog systems to navigate the chaos of multi-speaker interactions, achieving selective engagement in noisy environments.
Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens:<|respond|>,<|listen|>, and<|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in<|respond|>mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.
UW