Search papers, labs, and topics across Lattice.
This paper explores the development of continual motion-language agents that can both understand and generate human movement in dynamic environments, addressing the challenge of catastrophic forgetting when learning new motion concepts. By leveraging low-rank adaptation (LoRA) variants and a mixture-of-experts architecture, the authors demonstrate that their approach achieves near-zero forgetting while maintaining high quality in both motion-to-text and text-to-motion tasks. Key findings reveal that hard expert selection significantly enhances performance over soft blending, underscoring the importance of expert isolation in continual learning scenarios.
Near-zero forgetting in motion-language agents is achievable, but only with careful expert isolation during task transitions.
Motion-language agents must possess the bidirectional capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M). While foundational models have achieved strong performance in static settings, autonomous agents operating in dynamic environments must continuously incorporate new motion concepts -- such as novel athletic styles or specialized gestures -- without catastrophic forgetting of previously acquired skills. We investigate the stability-plasticity trade-off in bidirectional motion-language learning under sequential task exposure. Building on a frozen large language model backbone, we introduce low-rank adaptation (LoRA) variants designed to mitigate inter-task interference. We specifically propose mixture-of-experts architectures that utilize an autoencoder-based router to select task-specific experts at inference time, so that no task-label is needed. To evaluate these methods, we establish a reproducible five-task benchmark derived from HumanML3D through semantic clustering of motion descriptions. Our experimental results demonstrate near-zero forgetting across both M2T and T2M directions while maintaining high generation and captioning quality. Furthermore, we show that hard expert selection via routing significantly outperforms soft expert blending in quality metrics, indicating that preserving expert isolation is critical for maintaining performance in our continual learning setting. Finally, we observe that a divergence between token-level accuracy and downstream generation quality may occur, highlighting the need for more comprehensive evaluation protocols in future research on lifelong motion-language agents.
