Search papers, labs, and topics across Lattice.
This study explores the enhancement of brain-to-image decoding by augmenting small fMRI datasets with synthetic data generated by the TRIBE v2 model, which is pretrained on extensive fMRI responses. The authors demonstrate that using synthetic data can lead to a significant increase in image retrieval accuracy, achieving up to a 68% improvement in Top-10 accuracy compared to models trained solely on real data. Notably, they find that the optimal proportion of synthetic data varies by dataset, and in some cases, decoders trained exclusively on synthetic fMRI data can outperform random chance, indicating the potential for zero-shot decoding applications.
Augmenting fMRI datasets with synthetic data can yield a staggering 68% boost in image retrieval accuracy, challenging traditional limits of brain decoding.
Brain decoding is limited by the availability of labeled neural data, and remains challenging in low-data regimes. To address this issue, we investigate whether and when brain decoding can be boosted by augmenting small fMRI datasets with synthetic data generated by a pretrained model of fMRI responses to stimuli. We use TRIBE v2, a large encoding model pretrained on more than 1000 hours of fMRI responses to video, audio and language. For each dataset, we evaluate systematic grids that show how the performance of image decoders varies with the amount of synthetic data used for training. Our results, based on two datasets (the 7T fMRI Natural Scenes Dataset and 3T fMRI BOLD5000), show up to 68% improvement in Top-10 image-retrieval accuracy compared to decoders trained only on real data. Importantly, the proportion of augmented data required to reach a given image decoding performance needs to be adjusted depending on the data source. Surprisingly, image decoders trained exclusively on synthetic fMRI can perform above chance in some settings, suggesting that TRIBE v2 can support zero-shot brain-to-image decoding. Together, these results show how large-scale models of the fMRI responses to sight, sound and language may provide a foundation to improve the data efficiency for image decoding.
Meta AI