Search papers, labs, and topics across Lattice.
This paper introduces a multimodal RGB-D gait recognition framework for Parkinson's disease (PD) detection that enhances accuracy and interpretability. The framework uses dual YOLOv11 encoders for feature extraction from RGB and Depth data, followed by a Multi-Scale Local-Global Extraction (MLGE) module and Cross-Spatial Neck Fusion to improve spatial-temporal representation. A frozen Large Language Model (LLM) then translates fused visual embeddings and metadata into clinically relevant textual explanations, improving transparency and bridging the gap between visual recognition and clinical understanding.
Unlock clinically interpretable Parkinson's gait analysis by fusing RGB-D data with a frozen LLM, providing both high accuracy and clear visual-linguistic reasoning.
Accurate and interpretable gait analysis plays a crucial role in the early detection of Parkinsons disease (PD),yet most existing approaches remain limited by single-modality inputs, low robustness, and a lack of clinical transparency. This paper presents an explainable multimodal framework that integrates RGB and Depth (RGB-D) data to recognize Parkinsonian gait patterns under realistic conditions. The proposed system employs dual YOLOv11-based encoders for modality-specific feature extraction, followed by a Multi-Scale Local-Global Extraction (MLGE) module and a Cross-Spatial Neck Fusion mechanism to enhance spatial-temporal representation. This design captures both fine-grained limb motion (e.g., reduced arm swing) and overall gait dynamics (e.g., short stride or turning difficulty), even in challenging scenarios such as low lighting or occlusion caused by clothing. To ensure interpretability, a frozen Large Language Model (LLM) is incorporated to translate fused visual embeddings and structured metadata into clinically meaningful textual explanations. Experimental evaluations on multimodal gait datasets demonstrate that the proposed RGB-D fusion framework achieves higher recognition accuracy, improved robustness to environmental variations, and clear visual-linguistic reasoning compared with single-input baselines. By combining multimodal feature learning with language-based interpretability, this study bridges the gap between visual recognition and clinical understanding, offering a novel vision-language paradigm for reliable and explainable Parkinsons disease gait analysis. Code:https://github.com/manaralnaasan/RGB-D_parkinson-LLM
