Search papers, labs, and topics across Lattice.
This study introduces the Multimodal Voice Activity Projection (MM-VAP) framework, which enhances turn-taking prediction in social robots by integrating synchronized audio-visual inputs with a self-supervised future-projection objective. By leveraging pretrained audio-visual models and employing Low-Rank Adaptation, the framework effectively captures conversational dynamics through independent speaker encoding and inter-speaker attention mechanisms. Experimental results demonstrate significant improvements in predicting turn-taking events compared to existing baselines, highlighting the framework's potential for mediator applications in human-robot interaction.
MM-VAP significantly boosts turn-taking prediction accuracy in social robots by integrating audio-visual cues, outperforming existing models in key conversational scenarios.
Turn-taking prediction is a key requirement for social robots involved in human-human interaction, particularly in mediator settings, where the robot must anticipate conversational dynamics rather than merely react to pauses. This work presents a Multimodal Voice Activity Projection (MM-VAP) framework that extends the original audio-only VAP formulation to synchronized audio-visual inputs while preserving its self-supervised future-projection objective. The proposed approach builds on pretrained audio-visual backbones originally optimized for speech-related tasks and adapts them through Low-Rank Adaptation to the multimodal turn-taking problem. After independent speaker encoding, an inter-speaker attention stage models the relational dynamics required to project future voice activity. In addition, a semantic consistency loss is introduced to regularize the 256-state output space according to higher-level dialogue activity patterns. Experiments on NoXi and NoXi+J showed improvements over the current baselines, particularly for some turn-taking events. Additional evaluation on the Haru EDR corpus further supported the suitability of this direction for mediation-oriented human-robot interaction.
