Search papers, labs, and topics across Lattice.
This paper introduces Emotion-Cognition Multi-modal Captioning (ECMC), a new task for generating natural language descriptions of emotional and cognitive states from multi-modal data to enhance mental health understanding. They employ an encoder-decoder architecture with modality-specific encoders, a Q-former-based dual-stream BridgeNet for feature fusion, and contrastive learning to extract relevant features. Experiments show that ECMC outperforms existing models in generating emotion-cognition captions and improves assistive diagnosis and interpretability in mental health analysis.
LLMs can now generate detailed, interpretable descriptions of emotional and cognitive states from voices, faces, and text, opening new avenues for mental health analysis.
Emotional and cognitive factors are essential for understanding mental health disorders. However, existing methods often treat multi-modal data as classification tasks, limiting interpretability especially for emotion and cognition. Although large language models (LLMs) offer opportunities for mental health analysis, they mainly rely on textual semantics and overlook fine-grained emotional and cognitive cues in multi-modal inputs. While some studies incorporate emotional features via transfer learning, their connection to mental health conditions remains implicit. To address these issues, we propose ECMC, a novel task that aims at generating natural language descriptions of emotional and cognitive states from multi-modal data, and producing emotion-cognition profiles that improve both the accuracy and interpretability of mental health assessments. We adopt an encoder-decoder architecture, where modality-specific encoders extract features, which are fused by a dual-stream BridgeNet based on Q-former. Contrastive learning enhances the extraction of emotional and cognitive features. A LLaMA decoder then aligns these features with annotated captions to produce detailed descriptions. Extensive objective and subjective evaluations demonstrate that: 1) ECMC outperforms existing multi-modal LLMs and mental health models in generating emotion-cognition captions; 2) the generated emotion-cognition profiles significantly improve assistive diagnosis and interpretability in mental health analysis.
