Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces LaScA, a framework for predicting valence and arousal by using language models to condition handcrafted affect descriptors derived from facial geometry and acoustic features. These features are converted into natural language descriptions, which are then processed by a pretrained LM to generate semantic context embeddings that act as priors over affective dynamics. Experiments on Aff-Wild2 and SEWA datasets show LaScA improves accuracy compared to handcrafted-only and deep-embedding baselines, while maintaining feature transparency.
Unlock better affect prediction: Language models can condition interpretable, handcrafted features to outperform end-to-end deep learning, offering a transparent and efficient alternative.
Predicting affect in unconstrained environments remains a fundamental challenge in human-centered AI. While deep neural embeddings dominate contemporary approaches, they often lack interpretability and limit expert-driven refinement. We propose a novel framework that uses Language Models (LMs) as semantic context conditioners over handcrafted affect descriptors to model changes in Valence and Arousal. Our approach begins with interpretable facial geometry and acoustic features derived from structured domain knowledge. These features are transformed into symbolic natural-language descriptions encoding their affective implications. A pretrained LM processes these descriptions to generate semantic context embeddings that act as high-level priors over affective dynamics. Unlike end-to-end black-box pipelines, our framework preserves feature transparency while leveraging the contextual abstraction capabilities of LMs. We evaluate the proposed method on the Aff-Wild2 and SEWA datasets for affect change prediction. Experimental results show consistent improvements in accuracy for both Valence and Arousal compared to handcrafted-only and deep-embedding baselines. Our findings demonstrate that semantic conditioning enables interpretable affect modelling without sacrificing predictive performance, offering a transparent and computationally efficient alternative to fully end-to-end architectures