Search papers, labs, and topics across Lattice.
This paper presents SAC$^2$-Net, a novel architecture for multimodal micro-expression recognition that leverages Semantic Anchoring and Complementary-Consensus Fusion to address the challenges of cross-modal heterogeneity and spatially varying modality reliability. By aligning visual modalities with semantic anchors and employing a reliability-aware fusion strategy, the model effectively enhances the recognition of subtle facial movements associated with emotions. Experimental results demonstrate that SAC$^2$-Net achieves state-of-the-art performance across various benchmarks, highlighting its robustness in both coarse and fine-grained evaluations.
SAC$^2$-Net reveals that aligning motion-magnified and optical-flow representations through semantic anchors can significantly enhance micro-expression recognition performance.
Micro-expression recognition (MER) is challenging due to subtle facial movements, limited data, and the ambiguous relationship between Action Units (AUs) and emotion categories. Optical flow and motion magnification have been widely used to describe subtle facial dynamics from different perspectives: the former captures local motion displacement, while the latter amplifies weak appearance changes. In this work, we observe that these two modalities often exhibit asymmetric failure patterns: one modality may become noisy, distorted, or uninformative, while the other still preserves discriminative AU-related evidence. This phenomenon reveals their complementarity, but also raises two key challenges for fusion: cross-modal heterogeneity and spatially varying modality reliability. Motivated by this observation, we propose SAC$^2$-Net, a Semantic Anchoring and Complementary-Consensus Network for multimodal MER, which first aligns visual modalities with semantic anchors and then performs reliability-aware fusion. To reduce cross-modal heterogeneity before fusion, we introduce Semantic Anchoring Soft Alignment (SASA), which converts activated AUs into textual prompts and uses them as stable semantic anchors to align motion-magnified and optical-flow representations. Unlike hard contrastive learning, SASA constructs hierarchical AU-aware soft labels to preserve semantic proximity among samples with overlapping or anatomically related AU patterns. Based on the aligned representations, Complementary-Consensus Fusion (CCF) first repairs unreliable local evidence through complementary exchange and then enforces a shared spatial focus through consensus refinement. Extensive experiments on five MER benchmarks show that SAC$^2$-Net achieves state-of-the-art or highly competitive performance across coarse-grained, fine-grained, large-scale, and cross-dataset evaluation settings.
