Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces PRISM-AH, a novel framework for recognizing ambivalence and hesitancy (A/H) in health behavior change videos by leveraging multimodal data from facial, vocal, linguistic, and bodily signals. The framework aligns frozen encoders into short time windows and utilizes a lightweight streaming model to assess cross-modal dissonance while integrating participant metadata for enhanced prediction accuracy. Achieving a macro F1 score of 0.6133 on a test set of 525 videos, PRISM-AH significantly outperforms the zero-shot baseline of 0.2827, demonstrating its effectiveness in capturing complex emotional states over time.
A multimodal framework that accurately detects ambivalence and hesitancy in health behavior videos achieves a macro F1 score nearly double that of existing zero-shot methods.
Ambivalence and hesitancy (A/H) are conflicting affective states that precede the delay or abandonment of health behaviour change. Recognition of A/H at the video level is difficult, since the signal arises from disagreement across and within facial, vocal, linguistic, and bodily modalities, and manifests differently across individuals. The proposed PRISM-AH (Predictive Reasoning over Interacting Streams for Multimodal Ambivalence/Hesitancy Recognition), is a framework that treats A/H as a multimodal conflict that unfolds over time. Frozen vision, audio, and text encoders are aligned into short time windows and passed to a lightweight streaming model that scores cross-modal dissonance, predicts each next window to expose a hesitation surprise signal, discovers behaviour prototypes, and is conditioned on participant metadata. Dense window-level annotations supervise the model as an auxiliary objective, and the decision threshold is calibrated for macro F1. A knowledge-guided large language model then reasons over structured evidence using the expert cue taxonomy of the dataset, and its verdict is fused late only when validation performance improves. On the labelled public test partition of 525 videos, PRISM-AH attains a macro F1 of 0.6133, compared to the reported zero-shot baseline of 0.2827. The reasoning gain is validated to transfer from validation to the larger test partition.