Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to text-motion retrieval that leverages joint-angle motion images and token-patch late interaction to capture fine-grained local correspondences between text and 3D human motion. The method represents motion as structured pseudo-images based on joint-level local features, enabling the use of pre-trained Vision Transformers. By employing a MaxSim-based token-wise late interaction mechanism with Masked Language Modeling regularization, the model achieves robust and interpretable text-motion alignment. Experiments on HumanML3D and KIT-ML datasets demonstrate state-of-the-art retrieval performance and improved interpretability.
Ditch global embeddings: this text-to-motion retrieval method uses joint-angle motion images and token-patch late interaction for fine-grained alignment, outperforming SOTA while offering interpretability.
Text-motion retrieval aims to learn a semantically aligned latent space between natural language descriptions and 3D human motion skeleton sequences, enabling bidirectional search across the two modalities. Most existing methods use a dual-encoder framework that compresses motion and text into global embeddings, discarding fine-grained local correspondences, and thus reducing accuracy. Additionally, these global-embedding methods offer limited interpretability of the retrieval results. To overcome these limitations, we propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image, compatible with pre-trained Vision Transformers. For text-to-motion retrieval, we employ MaxSim, a token-wise late interaction mechanism, and enhance it with Masked Language Modeling regularization to foster robust, interpretable text-motion alignment. Extensive experiments on HumanML3D and KIT-ML show that our method outperforms state-of-the-art text-motion retrieval approaches while offering interpretable fine-grained correspondences between text and motion. The code is available in the supplementary material.
