Search papers, labs, and topics across Lattice.
Re$^2$MoGen addresses the challenge of open-vocabulary text-to-motion generation by using an LLM to generate initial motion plans, followed by physics-aware refinement. The approach uses Monte Carlo tree search to guide LLM reasoning for keyframe generation, then refines full-body poses using a human pose prior and dynamic temporal matching to fine-tune a motion generator. Post-training with a physics-aware reward function further enhances motion quality, leading to state-of-the-art performance in generating semantically consistent and physically plausible motions.
LLMs can generate physically plausible and semantically consistent motions from text descriptions previously unseen in training data, if you guide their reasoning with MCTS and refine the results with physics-aware RL.
Text-to-motion (T2M) generation aims to control the behavior of a target character via textual descriptions. Leveraging text-motion paired datasets, existing T2M models have achieved impressive performance in generating high-quality motions within the distribution of their training data. However, their performance deteriorates notably when the motion descriptions differ significantly from the training texts. To address this issue, we propose Re$^2$MoGen, a Reasoning and Refinement open-vocabulary Motion Generation framework that leverages enhanced Large Language Model (LLM) reasoning to generate an initial motion planning and then refine its physical plausibility via reinforcement learning (RL) post-training. Specifically, Re$^2$MoGen consists of three stages: We first employ Monte Carlo tree search to enhance the LLM's reasoning ability in generating reasonable keyframes of the motion based on text prompts, specifying only the root and several key joints'positions to ease the reasoning process. Then, we apply a human pose model as a prior to optimize the full-body poses based on the planned keyframes and use the resulting incomplete motion to supervise fine-tuning a pre-trained motion generator via a dynamic temporal matching objective, enabling spatiotemporal completion. Finally, we use post-training with physics-aware reward to refine motion quality to eliminate physical implausibility in LLM-planned motions. Extensive experiments demonstrate that our framework can generate semantically consistent and physically plausible motions and achieve state-of-the-art performance in open-vocabulary motion generation.
