Search papers, labs, and topics across Lattice.
This study evaluates the role of reasoning capabilities in multimodal large language models (MLLMs) for automatic dementia classification (ADC) from speech, revealing that naive reliance on text-based rationales can lead to hallucinations and poorer performance compared to LLM-free baselines. To address these challenges, the authors introduce DeTAiL, an adaptor-based framework that leverages the internal representations of reasoning MLLMs, resulting in enhanced classification accuracy. The proposed method consistently outperforms strong baselines across two distinct dementia datasets, highlighting the importance of effective model utilization in ADC tasks.
Relying on text-based rationales for dementia classification can actually degrade performance, revealing the need for more sophisticated approaches like DeTAiL.
Multimodal large language models (MLLMs) have emerged as a promising approach for improving the accuracy, transferability, and explainability of automatic dementia classification (ADC) systems from voice recordings. Yet it remains unclear whether their reasoning capabilities are beneficial for ADC, and how such capabilities should be leveraged. In this paper, we conduct a careful evaluation of reasoning MLLMs for ADC and show that naive strategies, such as relying on text-based rationales, can lead to hallucinated and inconsistent rationales for diagnosis and yield inferior ADC performance compared with LLM-free baselines. To overcome this limitation, we propose \textbf{De}mentia \textbf{T}hinker with Nonlinear \textbf{A}daptor and Re\textbf{i}nforcement \textbf{L}earning (DeTAiL), an adaptor-based framework that exploits the internal representations of reasoning MLLMs for improved dementia classification. Across two dementia datasets with distinct test formats and label granularities, DeTAiL consistently outperforms strong baselines and methods that rely on text-based rationales. Code and demo will be released upon acceptance.
MIT CSAIL