Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces TAMI, a novel multimodal fusion framework that addresses the challenges of temporal misalignment, missing data, and interpretability in mental health assessments for older adults with Mild Cognitive Impairment (MCI). By aligning various features—speech, language, facial expressions, and physiological data—on a shared timeline and accounting for missing values, TAMI significantly improves the accuracy of depression and anxiety detection. The framework achieved AUROC scores of 0.68 for depression and 0.69 for anxiety, with the most substantial performance gains attributed to fine-grained temporal alignment of multimodal features.
Fine-grained temporal alignment in multimodal analysis can boost depression and anxiety detection accuracy in older adults with MCI by over 10%.
Depression and anxiety in older adults with Mild Cognitive Impairment (MCI) are frequently underdiagnosed due to limited access to care. Multimodal analysis of remote clinical interviews is a scalable screening approach, but existing methods have three limitations. First, they do not correct temporal misalignment across multimodal features extracted at different resolutions, inducing spurious cross-modal associations. Second, remote recordings exhibit uneven modality dropout, but missing values are often zero-filled, making them indistinguishable from valid near-zero measurements. Finally, they do not jointly attribute predictions to modalities, questions, and interview moments, limiting fine-grained clinical interpretation. We propose a Temporally-Aligned, Missingness-Aware, Interpretable (TAMI) multimodal fusion framework. TAMI aligns speech, language, facial, and physiological features within question-answer segments on a shared timeline, encodes modality-level missingness over time, and conditions fusion on question context. In interviews with 49 older adults with MCI, TAMI achieved area under the receiver operating characteristic curve (AUROC) scores of 0.68 (depression) and 0.69 (anxiety). Fine-grained temporal alignment of multimodal features produced the largest performance gain ($Δ{\geq}0.1$). Multi-level interpretability analysis revealed that depression classification relied on eyegaze and open-ended questions, while anxiety classification depended on eyegaze and head pose, with attribution uniformly distributed across questions. Using only responses to the open-ended questions (5.1min), the depression model achieved an AUROC score of 0.67, which was not significantly different from using the full interview (19min) ($p>0.05$). Our findings support designing interview protocols centered on open-ended questions for depression screening in older adults with MCI.