Search papers, labs, and topics across Lattice.
To bypass the latency bottlenecks of bidirectional cross-modal transformers in speech emotion recognition, RAFM-SER++ couples a unidirectional Residual Attention Fusion Mechanism with a BYOL-inspired self-supervised alignment objective to project affective speech cues into semantic text representations. This asymmetric design eliminates heavy inter-modal computation, enabling high-throughput multimodal processing on resource-constrained edge and surveillance hardware. Evaluating on IEMOCAP and ESD benchmarks, the model slashes trainable parameters by over 60% and reaches 79.60 iterations/second while delivering state-of-the-art accuracy with 81.10% and 95.39% balanced accuracy, respectively.
Expensive bidirectional cross-attention is unnecessary for speech emotion recognition: routing acoustic cues unidirectionally into text representations slashes parameter count by 60% while outperforming heavier state-of-the-art transformers.
Recent multimodal Speech Emotion Recognition (SER) systems achieve high accuracy through interaction-heavy cross-modal transformers, but their computational cost limits deployment in latency-sensitive and resource-constrained surveillance systems. To address this challenge, we propose RAFM_SER++, a lightweight multimodal SER framework featuring an asymmetric Residual Attention Fusion Mechanism (RAFM). Rather than relying on computationally expensive bidirectional interactions, RAFM injects affective speech cues into semantic text representations through a one-directional residual attention pathway. Combined with a BYOL-inspired cross-modal alignment objective and attention-guided pooling, the proposed framework improves multimodal representation learning while maintaining low computational overhead. Experiments on the IEMOCAP and ESD benchmarks demonstrate that RAFM_SER++ consistently outperforms the HuBERT-Base baseline and achieves a superior accuracy-efficiency trade-off compared with the state-of-the-art MemoCMT. Specifically, RAFM_SER++ reduces trainable parameters by more than 60%, achieves faster inference (79.60 it/s), and attains BACC scores of 81.10% on IEMOCAP and 95.39% on ESD. These results indicate that lightweight asymmetric multimodal fusion is an effective alternative to interaction-heavy cross-modal transformers for real-time surveillance applications.
