Search papers, labs, and topics across Lattice.
MoCapAnything V2 introduces an end-to-end framework for monocular video motion capture of arbitrary skeletons, jointly optimizing Video-to-Pose and Pose-to-Rotation mappings. To resolve ambiguities in pose-to-rotation, they condition rotation prediction on a reference pose-rotation pair and rest pose, effectively defining the rotation coordinate system. Experiments demonstrate a reduction in rotation error from ~17 to ~10 degrees on Truebones Zoo and Objaverse, and to 6.54 degrees on unseen skeletons, with a ~20x speedup compared to mesh-based methods.
Achieve 20x faster motion capture for arbitrary skeletons with a new end-to-end framework that cuts rotation error nearly in half.
Recent methods for arbitrary-skeleton motion capture from monocular video follow a factorized pipeline, where a Video-to-Pose network predicts joint positions and an analytical inverse-kinematics (IK) stage recovers joint rotations. While effective, this design is inherently limited, since joint positions do not fully determine rotations and leave degrees of freedom such as bone-axis twist ambiguous, and the non-differentiable IK stage prevents the system from adapting to noisy predictions or optimizing for the final animation objective. In this work, we present the first fully end-to-end framework in which both Video-to-Pose and Pose-to-Rotation are learnable and jointly optimized. We observe that the ambiguity in pose-to-rotation mapping arises from missing coordinate system information: the same joint positions can correspond to different rotations under different rest poses and local axis conventions. To resolve this, we introduce a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system. This formulation turns rotation prediction into a well-constrained conditional problem and enables effective learning. In addition, our model predicts joint positions directly from video without relying on mesh intermediates, improving both robustness and efficiency. Both stages share a skeleton-aware Global-Local Graph-guided Multi-Head Attention (GL-GMHA) module for joint-level local reasoning and global coordination. Experiments on Truebones Zoo and Objaverse show that our method reduces rotation error from ~17 degrees to ~10 degrees, and to 6.54 degrees on unseen skeletons, while achieving ~20x faster inference than mesh-based pipelines. Project page: https://animotionlab.github.io/MoCapAnythingV2/
