Search papers, labs, and topics across Lattice.
This paper introduces In-Context Model Predictive Generation (ICMPG), a novel framework that combines language model planning with physical feedback to synthesize human motion from textual descriptions. By reformulating motion synthesis as a Model Predictive Control process, ICMPG utilizes a Context-Aware Motion Generation module to generate candidate motion sequences and a Model Predictive Generation module to evaluate these sequences against physical constraints and semantic fidelity. Experimental results demonstrate that ICMPG consistently outperforms existing methods, producing motions that are both physically plausible and semantically aligned with diverse commands in standard and zero-shot settings.
ICMPG achieves a groundbreaking balance between semantic fidelity and physical realism in motion synthesis, outperforming traditional methods in both standard and zero-shot scenarios.
Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and physical realism. Large language model (LLM)-based approaches can interpret diverse open-vocabulary instructions and compose high-level action plans, but they often generate motions that violate physical constraints. Physics-aware models improve realism through simulation or control, but they struggle with semantic complexity, fine-grained instructions, and novel concepts. To address this gap, we propose In-Context Model Predictive Generation (ICMPG), a framework that integrates language-model planning with inference-time physical feedback. ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules. The Context-Aware Motion Generation (CAMG) module uses an LLM as a planner to decompose textual commands and generate candidate motion sequences from motion tokens. The Model Predictive Generation (MPG) module evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence to guide subsequent generation steps. Unlike open-loop generation, this closed-loop refinement enables ICMPG to adapt motions to both the input semantics and the simulated physical environment without task-specific policy retraining. Extensive experiments across standard and zero-shot open-vocabulary settings show that ICMPG generalizes robustly to diverse commands and produces motions that are more physically plausible and semantically faithful than representative baselines on the evaluated benchmarks. The framework bridges semantic interpretation and physical simulation while remaining flexible enough to incorporate different LLM backbones, enabling more versatile and controllable text-driven motion synthesis.
Tsinghua AI