Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MuVAP, a novel multimodal framework that enhances turn-taking prediction in multiparty interactions by integrating acoustic data with facial tracking from a single camera and monaural audio. The method addresses the limitations of traditional setups by employing Role-Relative Projection to effectively manage the complexities of multiple speakers. Evaluations show that MuVAP significantly outperforms existing models on next-speaker prediction tasks, demonstrating its potential for real-world applications in human-robot interaction.
MuVAP achieves superior turn-taking predictions using just a single camera and microphone, revolutionizing how we approach multiparty interactions.
Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduce MuVAP, a causal multimodal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks, enabling speaker-aware turn-taking predictions from a monaural audio stream and a single camera view. To address the combinatorial complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state. Because existing audiovisual datasets contain disruptive editing cuts that break causal tracking, we introduce the Audio-Visual Conversation Corpus, a 31-hour dataset of unedited, single-camera multiparty conversations. Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks across two- and three-speaker settings.