Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
PolySLGen is introduced as an online framework for generating multimodal speaking and listening reactions in polyadic interactions, addressing the limitations of existing dyadic and single-modality approaches. The framework incorporates a pose fusion module and a social cue encoder to effectively model group dynamics by aggregating motion and social signals. Results demonstrate that PolySLGen generates contextually relevant and temporally coherent multimodal reactions, surpassing existing baselines in motion quality, speech alignment, speaking state prediction, and perceived realism.
AI can now generate more realistic and contextually appropriate reactions in group conversations, bridging the gap between dyadic interactions and complex social scenarios.
Human-like multimodal reaction generation is essential for natural group interactions between humans and embodied AI. However, existing approaches are limited to single-modality or speaking-only responses in dyadic interactions, making them unsuitable for realistic social scenarios. Many also overlook nonverbal cues and complex dynamics of polyadic interactions, both critical for engagement and conversational coherence. In this work, we present PolySLGen, an online framework for Polyadic multimodal Speaking and Listening reaction Generation. Given past conversation and motion from all participants, PolySLGen generates a future speaking or listening reaction for a target participant, including speech, body motion, and speaking state score. To model group interactions effectively, we propose a pose fusion module and a social cue encoder that jointly aggregate motion and social signals from the group. Extensive experiments, along with quantitative and qualitative evaluations, show that PolySLGen produces contextually appropriate and temporally coherent multi-modal reactions, outperforming several adapted and state-of-the-art baselines in motion quality, motion-speech alignment, speaking state prediction, and human-perceived realism.