Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a plug-and-play 2D Motion Interface that allows 3D-pretrained Motion Language Models (MoLMs) to process 2D motion inputs without requiring any modifications or fine-tuning. The approach addresses the challenge of accurately obtaining 3D motions from monocular videos, demonstrating performance on par with 3D inputs across various MoLMs while outperforming the training of MoLMs from scratch on 2D motions. Additionally, the authors present a new monocular real-world video motion evaluation dataset and a real-video adapter, highlighting the practical advantages of using 2D motions in real-world applications.
2D motion inputs can rival 3D data in performance, unlocking new possibilities for real-world applications of Motion Language Models without the need for extensive retraining.
Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.