Search papers, labs, and topics across Lattice.
This paper introduces TraMP-LLaMA, a multimodal framework that enhances facial expression quality assessment (FEQA) by simultaneously predicting severity scores and generating structured textual reports based on facial motion cues. By employing a decoupled instruction-tuning strategy, the framework minimizes task interference between severity prediction and language generation, leading to improved interpretability in clinical assessments, particularly for Parkinson's disease. Experimental results demonstrate that TraMP-LLaMA surpasses existing video-language baselines in report generation and achieves superior severity prediction performance, with a notable increase in Spearman's rank correlation by at least 4.39 percent over competing methods.
TraMP-LLaMA not only predicts severity scores but also generates interpretable reports from facial motion cues, bridging a critical gap in facial expression assessment.
Existing facial expression quality assessment (FEQA) methods typically produce only a severity score, without explicitly communicating the observable facial motion evidence that supports the prediction. This limits interpretability and makes it difficult to inspect the basis of model outputs in Parkinson's disease assessment. To address this gap, we propose TraMP-LLaMA, a unified multimodal framework that jointly predicts severity scores and generates structured textual reports from facial motion cues. The framework integrates RGB appearance and landmark trajectory cues, and adopts a decoupled instruction-tuning strategy to reduce task interference between severity prediction and language generation. To support this task, we further extend the PFED5 dataset with expert-guided textual motion descriptions and construct PFED5-plus. Experiments on PFED5-plus show that TraMP-LLaMA outperforms competitive video-language baselines in report generation and achieves the best severity prediction performance among the compared methods under joint multi-expression training, improving Spearman's rank correlation by at least 4.39 percent over all competing methods. The text annotations and code are available at https://github.com/shuchaoduan/TraMP-LLaMA.
