Search papers, labs, and topics across Lattice.
This paper introduces A4Mer, a nested latent Transformer architecture, to learn a hierarchical representation of human body movements from 3D pose data in a self-supervised manner. A4Mer decomposes pose sequences into Action Atoms (atomic joint movements) and then learns Action Motifs (temporal compositions of Atoms) via masked token prediction in latent spaces. The method is evaluated on a new large-scale dataset, AMD, and demonstrates improved performance on action recognition, motion prediction, and motion interpolation tasks.
Discovering reusable, semantic "Action Motifs" from human movement data unlocks significant gains in action recognition, motion prediction, and interpolation.
Effective human behavior modeling requires a representation of the human body movement that capitalizes on its compositionality. We propose a hierarchical representation consisting of Action Atoms that capture the atomic joint movements and Action Motifs that are formed by their temporal compositions and encode similar body movements found across different overall human actions. We derive A4Mer, a nested latent Transformer to learn this hierarchical representation from human pose data in a fully self-supervised manner. A4Mer splits a 3D pose sequence into variable-length segments and represents each segment as a single latent token (Action Atoms). Through bottom-up representation learning, temporal patterns composed of these Action Atoms, which capture meaningful temporal spans of reusable, semantic segments of body movements, naturally emerge (Action Motifs). A4Mer achieves this with a unified pretext task of masked token prediction in their respective latent spaces. We also introduce Action Motif Dataset (AMD), a large-scale dataset of multi-view human behavior videos with full SMPL annotations. We introduce a novel use of cameras by mounting them on the feet to achieve their frame-wise annotations despite frequent and heavy body occlusions. Experimental results demonstrate the effectiveness of A4Mer for extracting meaningful Action Motifs, which significantly benefit human behavior modeling tasks including action recognition, motion prediction, and motion interpolation.
