Search papers, labs, and topics across Lattice.
This paper introduces a Causal Modality-Invariant Representation (CmIR) learning framework to improve the robustness of multimodal affective computing models. CmIR disentangles each modality into causal invariant and environment-specific spurious representations using causal inference principles. Experiments on multimodal benchmarks demonstrate that CmIR achieves state-of-the-art performance, particularly on out-of-distribution and noisy data, indicating improved robustness and generalizability.
Multimodal models can now shrug off noisy or shifted data thanks to a new method that disentangles causal, stable signals from spurious correlations.
Multimodal affective computing aims to predict humans' sentiment, emotion, intention, and opinion using language, acoustic, and visual modalities. However, current models often learn spurious correlations that harm generalization under distribution shifts or noisy modalities. To address this, we propose a causal modality-invariant representation (CmIR) learning framework for robust multimodal learning. At its core, we introduce a theoretically grounded disentanglement method that separates each modality into `causal invariant representation' and `environment-specific spurious representation' from a causal inference perspective. CmIR ensures that the learned invariant representations retain stable predictive relationships with labels across different environments while preserving sufficient information from the raw inputs via invariance constraint, mutual information constraint, and reconstruction constraint. Experiments across multiple multimodal benchmarks demonstrate that CmIR achieves state-of-the-art performance. CmIR particularly excels on out-of-distribution data and noisy data, confirming its robustness and generalizability.
