Search papers, labs, and topics across Lattice.
This paper addresses the challenge of generating 3D human-human interactions (HHI) from text by modeling the underlying social structure that dictates interaction dynamics. The authors introduce a planner-executor paradigm, "Think with LLM, Move with Motion Skill," where a large language model (LLM) is used to plan the interaction phases and actor roles, while a motion executor translates this plan into physically plausible 3D motion. The proposed Solo-to-Social framework significantly enhances phase consistency, role alignment, and partner-aware coordination in generated interactions compared to previous methods.
LLMs can effectively decompose interactions into phases and roles, but struggle to generate dynamic, realistic motion without a structured approach.
Although text-to-motion generation has achieved strong progress in synthesizing realistic single-person motions from language, extending it to text-driven 3D human-human interaction (HHI) remains non-trivial, as HHI requires modeling the underlying \textbf{social structure} that governs phase progression, actor roles, and inter-actor coordination. In this paper, we formulate HHI generation as a social structure modeling and grounding problem: the model must first infer how an interaction unfolds and how the two actors coordinate their roles, and then realize this structure as continuous, physically plausible, and partner-aware 3D motion. To study how such structure should be modeled, we first examine the capability boundary of large language models (LLMs) for HHI generation. Our analysis shows that LLMs can \textit{think} by recovering phase decompositions and partner-aware roles, but cannot directly \textit{move}, as they fail to generate dynamic, physically plausible, and interaction-aware motion. This motivates our planner-executor paradigm, \textbf{Think with LLM, Move with Motion Skill}. The LLM planner converts implicit interaction semantics into motion-aligned social supervision by decomposing interactions into phases, assigning partner-aware actor roles, and aligning them with motion sequence. The motion executor then grounds the planned social structure into coordinated two-person motion by adapting a pretrained solo motion model with LoRA, previous-phase self-conditioning, and ego-relative partner conditioning. Together, our Solo-to-Social framework bridges social organization and motion realization, producing 3D HHI with improved phase consistency, role alignment, and partner-aware coordination.
NUS