Search papers, labs, and topics across Lattice.
BrainJanus introduces a unified model that integrates brain, vision, and language to address the limitations of existing unimodal approaches in neuroscience. By employing a Unified Brain Tokenizer to convert continuous neural dynamics into discrete tokens, the model facilitates seamless encoding and decoding across modalities using an All-in-One autoregressive architecture. Experimental results show that BrainJanus outperforms existing benchmarks while maintaining interpretable biological structures, indicating its potential as a versatile tool for brain modeling.
A unified model that seamlessly integrates brain, vision, and language reveals new insights into multimodal neural processing and outperforms traditional methods.
Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at https://github.com/HaitaoWuTJU/BrainJanus{GitHub}.
