Search papers, labs, and topics across Lattice.
The paper introduces Memory-guided Prototypical Co-occurrence Learning (MPCL) to address the limitations of existing mixed emotion recognition models that neglect valence consistency and structured correlations among coexisting emotions. MPCL fuses multi-modal signals using a multi-scale associative memory mechanism and constructs emotion-specific prototype memory banks to capture cross-modal semantic relationships, aligning them via prototype relation distillation. By incorporating a memory retrieval strategy, MPCL extracts semantic-level co-occurrence associations, leading to improved emotion distribution prediction, demonstrated through experiments on two public datasets.
By explicitly modeling emotion co-occurrence patterns, MPCL achieves state-of-the-art performance in mixed emotion recognition, outperforming existing methods that neglect the structured correlations among coexisting emotions.
Emotion recognition from multi-modal physiological and behavioral signals plays a pivotal role in affective computing, yet most existing models remain constrained to the prediction of singular emotions in controlled laboratory settings. Real-world human emotional experiences, by contrast, are often characterized by the simultaneous presence of multiple affective states, spurring recent interest in mixed emotion recognition as an emotion distribution learning problem. Current approaches, however, often neglect the valence consistency and structured correlations inherent among coexisting emotions. To address this limitation, we propose a Memory-guided Prototypical Co-occurrence Learning (MPCL) framework that explicitly models emotion co-occurrence patterns. Specifically, we first fuse multi-modal signals via a multi-scale associative memory mechanism. To capture cross-modal semantic relationships, we construct emotion-specific prototype memory banks, yielding rich physiological and behavioral representations, and employ prototype relation distillation to ensure cross-modal alignment in the latent prototype space. Furthermore, inspired by human cognitive memory systems, we introduce a memory retrieval strategy to extract semantic-level co-occurrence associations across emotion categories. Through this bottom-up hierarchical abstraction process, our model learns affectively informative representations for accurate emotion distribution prediction. Comprehensive experiments on two public datasets demonstrate that MPCL consistently outperforms state-of-the-art methods in mixed emotion recognition, both quantitatively and qualitatively.
Tsinghua AI