Search papers, labs, and topics across Lattice.
This paper introduces DyFADet+, a dual-stream RGB-skeleton framework designed for micro-gesture online recognition, which is crucial for understanding spontaneous emotions from subtle body movements in untrimmed videos. By employing a gated residual module to fuse multi-modal cues, the model effectively enhances the representation of gestures compared to traditional methods that rely on a single modality. The approach achieved an F1 score of 40.88 on the SMG dataset, securing the second position in the Micro-gesture Online Recognition track of the 4th EI-MiGA-IJCAI Challenge.
Fusing RGB and skeleton data with a gated residual module boosts micro-gesture recognition accuracy, revealing the power of multi-modal integration in emotion analysis.
Micro-gesture analysis attracts increasing attention for inferring spontaneous emotion from subtle body movements. Micro-gesture online recognition, which localizes and classifies each gesture instance in untrimmed videos, is a core task in the 4th EI-MiGA-IJCAI Challenge. Compared with typical temporal action detection, MGR emphasizes the localization and classification of actions, requiring the model to output the start time, end time, and category of each micro-gesture. Moreover, since micro-gestures are highly spontaneous, relying solely on a single modality makes it difficult to capture the complete and accurate multi-modal cues. In this work, we propose DyFADet+, which extends DyFADet into a dual-stream RGB-skeleton framework. In our model, both modalities are projected into shared multi-scale temporal embeddings and fused through a gated residual module, which adaptively injects skeleton motion into the RGB representation rather than using naive concatenation. Finally, these fused features are decoded by a Dynamic TAD head for online classification and boundary regression. On the SMG dataset, our method achieves an F1 score of 40.88, ranking 2nd in the Micro-gesture Online Recognition track.
