Search papers, labs, and topics across Lattice.
This paper introduces Dual-stream Semantic Enhancement (DuSE), a novel architecture for dynamic facial expression recognition (DFER) that incorporates cognitive principles of emotion perception. DuSE uses a dual-stream approach: a Hierarchical Temporal Prompt Cluster (HTPC) to model cognitive priming effects by aligning textual semantics with facial dynamics, and a Latent Semantic Emotion Aggregator (LSEA) to integrate sensory inputs with learned conceptual knowledge. Experiments on in-the-wild datasets demonstrate that DuSE achieves state-of-the-art performance in DFER while also improving model interpretability by emulating brain-like emotion processing.
Mimicking how the brain integrates language and context with facial cues unlocks state-of-the-art dynamic emotion recognition.
The human brain constructs emotional percepts not by processing facial expressions in isolation, but through a dynamic, hierarchical integration of sensory input with semantic and contextual knowledge. However, existing vision-based dynamic emotion modeling approaches often neglect emotion perception and cognitive theories. To bridge this gap between machine and human emotion perception, we propose cognition-inspired Dual-stream Semantic Enhancement (DuSE). Our model instantiates a dual-stream cognitive architecture. The first stream, a Hierarchical Temporal Prompt Cluster (HTPC), operationalizes the cognitive priming effect. It simulates how linguistic cues pre-sensitize neural pathways, modulating the processing of incoming visual stimuli by aligning textual semantics with fine-grained temporal features of facial dynamics. The second stream, a Latent Semantic Emotion Aggregator (LSEA), computationally models the knowledge integration process, akin to the mechanism described by the Conceptual Act Theory. It aggregates sensory inputs and synthesizes them with learned conceptual knowledge, reflecting the role of the hippocampus and default mode network in constructing a coherent emotional experience. By explicitly modeling these neuro-cognitive mechanisms, DuSE provides a more neurally plausible and robust framework for dynamic facial expression recognition (DFER). Extensive experiments on challenging in-the-wild benchmarks validate our cognition-centric approach, demonstrating that emulating the brain's strategies for emotion processing yields state-of-the-art performance and enhances model interpretability.
