Search papers, labs, and topics across Lattice.
This paper introduces a multimodal emotion recognition framework using a dual-branch Transformer with safe cross-attention and modality dropout to handle partial occlusions and missing modalities in real-world environments. The framework dynamically fuses visual and audio representations, leveraging audio when visual cues are absent. The method achieves an accuracy of 60.79% and an F1-score of 0.5029 on the Aff-Wild2 validation set by also employing focal loss and a sliding-window soft voting strategy to address class imbalance and reduce classification jitter.
A dual-branch Transformer with safe cross-attention overcomes missing visual cues in emotion recognition by dynamically relying on audio, achieving state-of-the-art results on Aff-Wild2.
Emotion recognition in real-world environments is hindered by partial occlusions, missing modalities, and severe class imbalance. To address these issues, particularly for the Affective Behavior Analysis in-the-wild (ABAW) Expression challenge, we propose a multimodal framework that dynamically fuses visual and audio representations. Our approach uses a dual-branch Transformer architecture featuring a safe cross-attention mechanism and a modality dropout strategy. This design allows the network to rely on audio-based predictions when visual cues are absent. To mitigate the long-tail distribution of the Aff-Wild2 dataset, we apply focal loss optimization, combined with a sliding-window soft voting strategy to capture dynamic emotional transitions and reduce frame-level classification jitter. Experiments demonstrate that our framework effectively handles missing modalities and complex spatiotemporal dependencies, achieving an accuracy of 60.79% and an F1-score of 0.5029 on the Aff-Wild2 validation set.
