Search papers, labs, and topics across Lattice.
This paper introduces a Transformer-based architecture for audio-visual emotion recognition (AVER) that explicitly addresses frame-rate mismatch between audio and video. The core innovation is Temporally-aligned Rotary Position Embeddings (TaRoPE) to implicitly synchronize audio and video tokens within a multimodal self-attention encoder. A Cross-Temporal Matching (CTM) loss further encourages consistency between temporally aligned pairs, leading to improved performance on CREMA-D and RAVDESS datasets compared to existing methods.
Explicitly aligning audio and video streams in a multimodal Transformer boosts emotion recognition, showing that ignoring frame-rate differences hurts performance.
Audio-visual emotion recognition (AVER) methods typically fuse utterance-level features, and even frame-level attention models seldom address the frame-rate mismatch across modalities. In this paper, we propose a Transformer-based framework focusing on the temporal alignment of multimodal features. Our design employs a multimodal self-attention encoder that simultaneously captures intra- and inter-modal dependencies within a shared feature space. To address heterogeneous sampling rates, we incorporate Temporally-aligned Rotary Position Embeddings (TaRoPE), which implicitly synchronize audio and video tokens. Furthermore, we introduce a Cross-Temporal Matching (CTM) loss that enforces consistency among temporally proximate pairs, guiding the encoder toward better alignment. Experiments on CREMA-D and RAVDESS datasets demonstrate consistent improvements over recent baselines, suggesting that explicitly addressing frame-rate mismatch helps preserve temporal cues and enhances cross-modal fusion.
