Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study explores the use of an end-to-end deep learning framework for visual semantic decoding from electrocorticography (ECoG) data, specifically predicting visual categories from video stimuli. Utilizing a dataset from 17 participants with drug-resistant epilepsy and employing fewer than 50 training samples per category, the research evaluates various deep learning architectures and identifies a Transformer-based encoder as the most effective approach. Key findings reveal that the decoding performance is significantly influenced by specific brain regions, particularly the early visual cortex and ventral stream, while also demonstrating the model's interpretability in relation to established neuroscience principles.
An end-to-end deep learning model can decode visual categories from ECoG data with surprising accuracy, revealing critical insights into the brain's processing of visual stimuli.
ECoG-based visual semantic decoding enables inference of semantic interpretation of visual perception from complex, noisy brain activity. This study examines the feasibility of visual semantic decoding using an end-to-end deep learning framework using electrocorticography (ECoG). Specifically, the decoding task is to predict visual categories from video stimuli using time-series neural inputs. A previously collected ECoG dataset from participants ($n=17$) with drug-resistant epilepsy is used for analysis. With fewer than 50 training samples per visual category, this study evaluates multiple deep learning approaches, artificial neural network architectures, and frequency-band filtered inputs. The best-performing approach is analyzed to shed light on the discriminative information it relies on across spectral, temporal, and cortical dimensions. The selected decoding system uses mixup augmentation, a Transformer-based encoder, and high-gamma (80-150 Hz) inputs with a 900 ms post-stimulus window. Further analysis shows that early visual cortex (V2-V4), ventral stream visual cortex, MT+ complex with neighbouring visual areas, and lateral temporal cortex contributed substantially to decoding performance. This study demonstrates that an end-to-end deep learning framework can yield promising decoding performance from dynamic visual stimuli without handcrafted features, while the model behavior remains interpretable through spectral, temporal, and cortical dimensions, which are broadly consistent with established neuroscience knowledge.