Search papers, labs, and topics across Lattice.
This paper introduces DualAlign, a two-stage multi-modal fusion framework designed to enhance Action Quality Assessment (AQA) by addressing the challenges of cross-modal misalignment and limited dataset diversity. By maximizing shared structural information across RGB video, optical flow, and skeleton modalities, and subsequently incorporating textual semantics, the framework stabilizes visual representations and improves the assessment of movement quality. Experimental results demonstrate that DualAlign achieves a 21.16% improvement in average correlation on the newly introduced MM--JDM dataset compared to state-of-the-art methods, while also showing robustness under conditions of missing modalities and label scarcity.
DualAlign boosts action quality assessment accuracy by over 21% through innovative multi-modal fusion and adaptive alignment techniques.
Action Quality Assessment (AQA) aims to evaluate how well a person performs a movement, which is essential in applications such as sports scoring, skill assessment, and healthcare. However, unimodal approaches often struggle to capture subtle cues of movement quality in real-world settings. Although multi-modal inputs provide complementary information, existing methods still face two major challenges: heterogeneous modalities often lead to cross-modal misalignment and unstable fusion, and reliable multi-modal annotation is costly, resulting in limited dataset diversity. To address these challenges, we propose DualAlign, a two-stage multi-modal fusion framework with adaptive alignment. The framework first constructs a coherent visual representation by maximizing shared structural information across RGB video, optical flow, and skeleton modalities. Textual semantics are then incorporated after visual stabilization, allowing high-level descriptions to complement rather than distort the underlying visual manifold. To evaluate the framework under realistic multi-modal conditions, we introduce MM--JDM, a movement-quality assessment dataset integrating RGB videos, optical flow, skeleton sequences, and structured text. MM--JDM naturally exhibits modality noise, class imbalance, and label scarcity, making it a challenging benchmark for studying multi-modal fusion and alignment. Extensive experiments show that DualAlign improves average correlation on MM--JDM by 21.16% over the state-of-the-art methods and achieves gains of 3.53% and 5.95% on the RG and Fis-V benchmarks, respectively. DualAlign also remains robust under missing-modality and label-scarce conditions.
