Search papers, labs, and topics across Lattice.
This paper introduces MyoMechanix, a multimodal ecosystem designed for biomechanically-grounded action quality assessment (AQA) that integrates motion data with muscle activity, addressing the limitations of existing visual-only methods. The dataset comprises over 7,500 samples of 20 actions from 38 subjects, featuring synchronized multiview RGB video, 3D pose data, and sEMG signals, making it the largest AQA benchmark to date. Key innovations include the Fitness Knowledge Graph for structured feedback and the CUBIST engine for fine-grained error attribution, resulting in state-of-the-art performance and enhanced interpretability in skilled activity understanding.
CUBIST achieves state-of-the-art results in action quality assessment by leveraging multimodal data for precise error attribution and feedback generation.
Existing action quality assessment (AQA) datasets and methods rely primarily on visual inputs such as RGB and pose, overlooking physiological dynamics such as muscle mechanics and often modeling actions as monolithic patterns. These limitations hinder fine-grained, biomechanically grounded feedback. We introduce MyoMechanix, a multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity. Expert-annotated, it contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB video, 3D pose, sEMG, and additional physiological signals, forming the largest multimodal AQA benchmark to date. We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedback, enabling compositional scoring and interpretable assessment. Building on these representations, we develop CUBIST (Compositional Ontological Reasoning Engine), which performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation. We also establish MyoMechanix-AQA, MyoMechanix-VideoQA, and a novel MyoMechanix-Video2EMG task. Experiments show that multimodal sensing and structured representations improve performance, interpretability, and error attribution, with CUBIST achieving state-of-the-art results; VideoQA enhances language-grounded action understanding; and Video2EMG suggests video-based alternatives to costly EMG sensing. MyoMechanix advances skilled activity understanding toward biomechanically grounded, multimodal, and compositional reasoning for Physical AI applications in fitness, rehabilitation, healthcare, and machine learning. Project page: https://haoyin116.github.io/MyoMechanix/
