Search papers, labs, and topics across Lattice.
This study explores the capacity of Transformer embeddings to reflect the geometric structure of Russell's circumplex model of affect, addressing the opacity of latent spaces in affective computing. By conducting two experiments with text and speech data, the authors demonstrate that multimodal fusion of these embeddings achieves perfect alignment with the valence-arousal topology and accurately positions fine-grained emotion terms in a zero-shot context. The findings suggest that the structure of human emotions is inherently captured in the latent representations of deep learning models, providing a robust framework for validating emotion models in AI.
Multimodal fusion of text and audio reveals that deep learning embeddings perfectly encode the geometric structure of human emotions, challenging the notion that such structures are merely artifacts of human labeling.
Affective computing increasingly relies on deep learning to represent emotions, yet latent spaces often remain opaque, high-dimensional black boxes. This paper investigates whether Transformers' embeddings recover the geometric regularities of Russell's circumplex model. We unify two complementary experiments testing the hypothesis that, after training models on text and speech, their resulting latent spaces encode a topology consistent with valence-arousal and reproduce human-like neighborhood relations. Specifically, we evaluate deep representations extracted from Transformer-based text (RoBERTa) and speech (wav2vec 2.0) encoders, along with a multimodal Transformer fusion architecture, across naturalistic datasets like MSP-Podcast and controlled LLM-generated stimuli. Our analysis reveals that multimodal fusion of text and audio yields perfect topological alignment with Russell's primary emotion ordering. Furthermore, in a zero-shot setting using generic text embeddings, projected fine-grained emotion terms fall close to their established human-mapped coordinates. Our contribution is a novel, data-driven framework for validating emotion models, demonstrating that Russell's circumplex structure is intrinsically encoded in the embeddings of these modalities rather than being solely an artifact of human labeling, thereby bridging the gap between psychological theory and representation learning.
