Search papers, labs, and topics across Lattice.
This paper introduces the Primitive Memory Distillation (PriMD) framework, which enhances Multimodal Emotion Recognition (MER) by focusing on the intra-modal differences in recoverability of information rather than treating modalities holistically. By disentangling shared semantics from modality-specific representations and constructing learnable semantic primitives, PriMD allows for dynamic retrieval of information when modalities are missing. Extensive experiments demonstrate that PriMD achieves state-of-the-art performance and robustness across various scenarios of missing modalities, addressing the instability of traditional methods.
PriMD revolutionizes multimodal emotion recognition by enabling robust performance even when critical data modalities are missing.
Multimodal Emotion Recognition (MER) systems often suffer from missing modalities in real-world scenarios. Existing methods usually generate, align, or distill missing modalities as a whole, overlooking the heterogeneous nature of the information carried by each modality. Such holistic treatment mixes inferable shared semantics with uncertain modality-specific details, yielding unstable representations and degrading robustness. To address this issue, we propose the Primitive Memory Distillation (PriMD) framework. Unlike existing methods, PriMD takes an intra-modal perspective and focuses on how different types of information within a modality differ in recoverability within each modality. PriMD first disentangles cross-modal shared semantics from modality-specific representations, and then discretizes the latter into learnable semantic primitives to construct modality-specific memory banks. When modalities are missing, PriMD is a teacher-student framework that the student model uses the shared semantics of available modalities as queries to dynamically retrieve primitives. It compensates for missing modality-specific information within a constrained memory space and aligns with the teacher model. Extensive experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate that PriMD achieves state-of-the-art performance and consistently stronger robustness across a wide range of missing-modality settings, while mitigating the instability caused by holistic feature inference. Our code and project website are available at https://github.com/JiaqiZhang-Sengoku/PriMD and https://jiaqizhang-sengoku.github.io/PriMD/, respectively.
NUS