Search papers, labs, and topics across Lattice.
The authors introduce NOBEL, a neuro-omni-modal brain-encoding large language model, to unify heterogeneous EEG/MEG and fMRI signals within a shared semantic embedding space. NOBEL employs a unified encoder for EEG/MEG and a dual-path strategy for fMRI, aligning these signals with external sensory stimuli before leveraging an LLM backbone. Experiments demonstrate NOBEL's strong performance on standard single-modal tasks and improved decoding accuracy through synergistic fusion of electromagnetic and metabolic signals, particularly in stimulus-aware decoding tasks using the NSD and HAD datasets.
Forget separate pipelines for EEG, MEG, and fMRI data – this LLM fuses them all into a single semantic space, unlocking more accurate brain decoding.
Deciphering brain function through non-invasive recordings requires synthesizing complementary high-frequency electromagnetic (EEG/MEG) and low-frequency metabolic (fMRI) signals. However, despite their shared neural origins, extreme discrepancies have traditionally confined these modalities to isolated analysis pipelines, hindering a holistic interpretation of brain activity. To bridge this fragmentation, we introduce \textbf{NOBEL}, a \textbf{n}euro-\textbf{o}mni-modal \textbf{b}rain-\textbf{e}ncoding \textbf{l}arge language model (LLM) that unifies these heterogeneous signals within the LLM's semantic embedding space. Our architecture integrates a unified encoder for EEG and MEG with a novel dual-path strategy for fMRI, aligning non-invasive brain signals and external sensory stimuli into a shared token space, then leverages an LLM as a universal backbone. Extensive evaluations demonstrate that NOBEL serves as a robust generalist across standard single-modal tasks. We also show that the synergistic fusion of electromagnetic and metabolic signals yields higher decoding accuracy than unimodal baselines, validating the complementary nature of multiple neural modalities. Furthermore, NOBEL exhibits strong capabilities in stimulus-aware decoding, effectively interpreting visual semantics from multi-subject fMRI data on the NSD and HAD datasets while uniquely leveraging direct stimulus inputs to verify causal links between sensory signals and neural responses. NOBEL thus takes a step towards unifying non-invasive brain decoding, demonstrating the promising potential of omni-modal brain understanding.
Tsinghua AI