Search papers, labs, and topics across Lattice.
This paper introduces Uncertainty-aware Robust Multimodal Fusion (URMF) to address the issue of unreliable modalities in multimodal sarcasm detection (MSD). URMF models modality reliability by parameterizing text, image, and interaction-aware latent representations as learnable Gaussian posteriors, estimating aleatoric uncertainty. By dynamically regulating modality contributions based on this uncertainty during fusion, URMF achieves more robust joint representations and outperforms strong baselines on MSD benchmarks.
Ignoring modality uncertainty in multimodal sarcasm detection hurts performance, but modeling it with a Gaussian posterior and using it to regulate fusion achieves state-of-the-art results.
Multimodal sarcasm detection (MSD) aims to identify sarcastic intent from semantic incongruity between text and image. Although recent methods have improved MSD through cross-modal interaction and incongruity reasoning, they often assume that all modalities are equally reliable. In real-world social media, however, textual content may be ambiguous and visual content may be weakly relevant or even irrelevant, causing deterministic fusion to introduce noisy evidence and weaken robust reasoning. To address this issue, we propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework that explicitly models modality reliability during interaction and fusion. URMF first employs multi-head cross-attention to inject visual evidence into textual representations, followed by multi-head self-attention in the fused semantic space to enhance incongruity-aware reasoning. It then performs unified unimodal aleatoric uncertainty modeling over text, image, and interaction-aware latent representations by parameterizing each modality as a learnable Gaussian posterior. The estimated uncertainty is further used to dynamically regulate modality contributions during fusion, suppressing unreliable modalities and yielding a more robust joint representation. In addition, we design a joint training objective integrating task supervision, modality prior regularization, cross-modal distribution alignment, and uncertainty-driven self-sampling contrastive learning. Experiments on public MSD benchmarks show that URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines, demonstrating the effectiveness of uncertainty-aware fusion for improving both accuracy and robustness.
