Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the task of generating listener body motions reactive to speaker utterances, addressing the challenge of modeling non-deterministic human reactions. To facilitate this, the authors present ReactMotionNet, a large-scale dataset pairing speaker utterances with multiple listener motion candidates annotated for appropriateness. They then propose ReactMotion, a generative framework trained with preference-based objectives that jointly models text, audio, emotion, and motion, outperforming retrieval and LLM-based baselines in generating natural and appropriate listener motions.
Finally, realistic and diverse listener reactions to speech can be automatically generated, moving beyond simple retrieval or LLM-driven approaches.
In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.