Search papers, labs, and topics across Lattice.
The paper introduces AULLM++, a framework for micro-expression action unit (AU) detection that leverages large language models (LLMs) to reason about visual features and AU relationships. AULLM++ uses a Multi-Granularity Evidence-Enhanced Fusion Projector (MGE-EFP) to create a Content Token (CT) from visual features and a Relation-Aware AU Graph Neural Network (R-AUGNN) to generate an Instruction Token (IT) encoding AU relationships, which are then combined in a textual prompt for the LLM. The framework achieves state-of-the-art performance on micro-expression benchmarks and demonstrates improved cross-domain generalization through the use of Counterfactual Consistency Regularization (CCR).
LLMs can significantly boost micro-expression recognition by reasoning about subtle facial muscle movements when guided by structured visual and relational prompts.
Micro-expression Action Unit (AU) detection identifies localized AUs from subtle facial muscle activations, providing a foundation for decoding affective cues. Previous methods face three key limitations: (1) heavy reliance on low-density visual information, rendering discriminative evidence vulnerable to background noise; (2) coarse-grained feature processing that misaligns with the demand for fine-grained representations; and (3) neglect of inter-AU correlations, restricting the parsing of complex expression patterns. We propose AULLM++, a reasoning-oriented framework leveraging Large Language Models (LLMs), which injects visual features into textual prompts as actionable semantic premises to guide inference. It formulates AU prediction into three stages: evidence construction, structure modeling, and deduction-based prediction. Specifically, a Multi-Granularity Evidence-Enhanced Fusion Projector (MGE-EFP) fuses mid-level texture cues with high-level semantics, distilling them into a compact Content Token (CT). Furthermore, inspired by micro- and macro-expression AU correspondence, we encode AU relationships as a sparse structural prior and learn interaction strengths via a Relation-Aware AU Graph Neural Network (R-AUGNN), producing an Instruction Token (IT). We then fuse CT and IT into a structured textual prompt and introduce Counterfactual Consistency Regularization (CCR) to construct counterfactual samples, enhancing the model's generalization. Extensive experiments demonstrate AULLM++ achieves state-of-the-art performance on standard benchmarks and exhibits superior cross-domain generalization.
