Search papers, labs, and topics across Lattice.
This paper introduces a multi-modal framework designed for recognizing micro-gestures (MGs) in untrimmed videos, addressing challenges such as low signal-to-noise ratios and long-tailed class distributions. The authors implement a saliency-guided extraction pipeline that combines keypoint skeleton coordinates, 3D heatmaps, and RGB features, while employing a novel Cross-Modal Pseudo-Labeling strategy to enhance cross-subject generalization. Their approach achieves a competitive F1-score of 68.13%, placing them fourth in the MiGA-IJCAI Challenge, showcasing significant improvements in single-modal robustness and tail class recognition.
Cross-Modal Pseudo-Labeling boosts micro-gesture recognition performance, bridging the gap between subjects and enhancing model robustness.
Micro-gestures (MGs) are spontaneous and subtle body movements that frequently convey hidden human emotions. Recognizing MGs in untrimmed videos remains highly challenging due to their extremely low signal-to-noise ratio, severe long-tailed class distribution, and the inherent domain shift encountered in cross-subject evaluation scenarios. In this paper, we propose a comprehensive multi-modal framework for Track 1 of the 4th MiGA-IJCAI Challenge. To capture fine-grained representations, we design a saliency-guided multi-modal extraction pipeline integrating 68-keypoint skeleton joint coordinates, 3D heatmap volumes, and high-resolution RGB visual features. We introduce a gentle square-root smoothed weighting mechanism paired with an Orthogonal Semantic Embedding Loss to protect tail classes without compromising overall recognition capabilities. More importantly, to bridge the cross-subject generalization gap, we propose a Cross-Modal Pseudo-Labeling (CMPL) strategy for unsupervised domain adaptation, which significantly boosts single-modal robustness. A temperature-scaled soft-voting mechanism is finally utilized to alleviate overconfidence during late fusion. Extensive experiments demonstrate that our framework achieves a competitive F1-score of 68.13\%, securing the 4th place.
