Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce NOEMA, a temporal convolutional decoder that reconstructs perceived naturalistic speech from fMRI by aligning noisy, temporally smeared brain activity with multimodal stimulus embeddings derived from text, audio, and speech representations. NOEMA achieves over 42% retrieval accuracy within the top 10 candidates out of 2,000 for held-out queries on two naturalistic speech datasets, significantly outperforming baseline decoders. Ablations confirm the critical roles of both temporal modeling and complementary stimulus representations in this performance gain.
Brain activity can decode naturalistic speech segments with 42% accuracy within the top 10 of 2,000 options, a feat achieved by aligning fMRI with multimodal stimulus embeddings.
Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.