Search papers, labs, and topics across Lattice.
This paper introduces an innovative audio-text recognition system designed specifically for detecting ambivalence and hesitancy in videos, utilizing a combination of prosodic audio descriptors, emotion-oriented text embeddings, and handcrafted psycholinguistic features. By employing temporal cross-attention and gated multiple-instance learning pooling, the model effectively captures the nuances of uncertainty and hedging in speech. The ensemble approach achieved a notable average precision of 0.875 and a macro-F1 score of 0.72 on the public development set, showcasing its effectiveness in this challenging domain.
Achieving an impressive 87.5% precision in recognizing ambivalence and hesitancy, this system redefines how we analyze emotional nuances in audio-text data.
We present an audio-text system for the Ambivalence/Hesitancy Video Recognition Challenge of the 11th ABAW Competition. The method excludes visual frames and represents each video as overlapping 5-second windows aligned with transcript timestamps. Each window combines a 320-dimensional prosodic audio descriptor, a 768-dimensional emotion-oriented RoBERTa embedding, and 74 handcrafted features capturing uncertainty, hedging, and attitudinal conflict. Audio and text are fused via temporal cross-attention, while support features are injected prior to gated multiple-instance learning (MIL) pooling to modulate the window's importance. Predictions from five independently initialized models are averaged. On the labeled public development set, the ensemble achieved an average precision of 0.875 and a macro-F1 of 0.72. Our source code is publicly available at https://github.com/Liga-de-IA-PUCPR/abaw-11-ah-challenge/.
