Search papers, labs, and topics across Lattice.
This paper introduces CLIP-AU, a method that leverages Action Units (AUs) as structured textual prompts within CLIP to model fine-grained facial expressions for emotion recognition, and CLIP-AUTT, a test-time personalization method that dynamically adapts AU prompts to videos from unseen subjects. CLIP-AUTT uses entropy-guided temporal window selection with prompt tuning to achieve subject-specific adaptation while preserving temporal consistency, avoiding the need for CLIP fine-tuning or LLM-generated text supervision. Experiments on three video datasets demonstrate that CLIP-AU and CLIP-AUTT outperform state-of-the-art CLIP-based methods for fine-grained emotion recognition.
Forget noisy LLM-generated prompts: this method uses interpretable Action Units to guide CLIP for personalized, fine-grained video emotion recognition, achieving state-of-the-art results.
Personalization in emotion recognition (ER) is essential for an accurate interpretation of subtle and subject-specific expressive patterns. Recent advances in vision-language models (VLMs) such as CLIP demonstrate strong potential for leveraging joint image-text representations in ER. However, CLIP-based methods either depend on CLIP's contrastive pretraining or on LLMs to generate descriptive text prompts, which are noisy, computationally expensive, and fail to capture fine-grained expressions, leading to degraded performance. In this work, we leverage Action Units (AUs) as structured textual prompts within CLIP to model fine-grained facial expressions. AUs encode the subtle muscle activations underlying expressions, providing localized and interpretable semantic cues for more robust ER. We introduce CLIP-AU, a lightweight AU-guided temporal learning method that integrates interpretable AU semantics into CLIP. It learns generic, subject-agnostic representations by aligning AU prompts with facial dynamics, enabling fine-grained ER without CLIP fine-tuning or LLM-generated text supervision. Although CLIP-AU models fine-grained AU semantics, it does not adapt to subject-specific variability in subtle expressions. To address this limitation, we propose CLIP-AUTT, a video-based test-time personalization method that dynamically adapts AU prompts to videos from unseen subjects. By combining entropy-guided temporal window selection with prompt tuning, CLIP-AUTT enables subject-specific adaptation while preserving temporal consistency. Our extensive experiments on three challenging video-based subtle ER datasets, BioVid, StressID, and BAH, indicate that CLIP-AU and CLIP-AUTT outperform state-of-the-art CLIP-based FER and TTA methods, achieving robust and personalized subtle ER.
