Search papers, labs, and topics across Lattice.
This paper introduces M3D-BFS, a multi-stage dynamic fusion strategy for integrating structural and functional connectivity in brain networks. The key innovation is a sample-adaptive mixture-of-experts (MoE) architecture that dynamically adjusts its computation based on the input sample, unlike static fusion methods. By employing a three-stage training process and a multi-modal disentanglement loss, M3D-BFS achieves state-of-the-art performance on real-world brain network datasets.
Brain network analysis gets a dynamic upgrade: M3D-BFS adaptively fuses multi-modal data based on individual samples, boosting performance beyond static methods.
Multi-modal fusion is of great significance in neuroscience which integrates information from different modalities and can achieve better performance than uni-modal methods in downstream tasks. Current multi-modal fusion methods in brain networks, which mainly focus on structural connectivity (SC) and functional connectivity (FC) modalities, are static in nature. They feed different samples into the same model with identical computation, ignoring inherent difference between input samples. This lack of sample adaptation hinders model's further performance. To this end, we innovatively propose a multi-stage dynamic fusion strategy (M3D-BFS) for sample-adaptive multi-modal brain network analysis. Unlike other static fusion methods, we design different mixture-of-experts (MoEs) for uni- and multi-modal representations where modules can adaptively change as input sample changes during inference. To alleviate issue of MoE where training of experts may be collapsed, we divide our method into 3 stages. We first train uni-modal encoders respectively, then pretrain single experts of MoEs before finally finetuning the whole model. A multi-modal disentanglement loss is designed to enhance the final representations. To the best of our knowledge, this is the first work for dynamic fusion for multi-modal brain network analysis. Extensive experiments on different real-world datasets demonstrates the superiority of M3D-BFS.
