Search papers, labs, and topics across Lattice.
This study introduces EEGAlign, a novel framework that simultaneously aligns electroencephalography (EEG) data with text and audio features to decode Chinese sentences from non-invasive EEG recordings. By leveraging contrastive learning with BGE-M3 text embeddings and wav2vec 2.0 speech features, EEGAlign achieves state-of-the-art performance in sentence classification, with Top-1 accuracy reaching 82.37% for Reading Aloud EEG. The findings highlight the complementary nature of text and audio alignment in enhancing decoding accuracy, marking a significant advancement in non-invasive neural communication for individuals with speech impairments.
Achieving over 82% accuracy in decoding large-vocabulary Chinese sentences from non-invasive EEG could revolutionize communication for individuals with severe speech impairments.
Decoding speech information directly from scalp electroencephalography (EEG) into text provides a potential non-invasive neural communication pathway for individuals with severe speech and motor impairments. Compared with invasive approaches such as electrocorticography, EEG is safer and more widely deployable, yet substantially more challenging to decode.This challenge is exacerbated for Chinese sentence decoding, which must handle a high-dimensional output space with thousands of characters, severe inter-subject variability, and low signal-to-noise ratios for text alignment.Existing methods commit to a single supervisory axis---either text semantics or audio acoustic features---yet neither can simultaneously satisfy the demands of sentence-level discriminability and fine-grained temporal resolution required for large-vocabulary Chinese decoding. We introduce EEGAlign, a novel parameter-efficient framework that jointly aligns EEG with two axes---text alignment with BGE-M3 text embeddings and audio alignment with wav2vec~2.0 speech features via contrastive learning followed by CTC character-sequence decoding. On ChineseEEG-2 data, EEGAlign yields state-of-the-art closed-set sentence classification performance, reaching up to 82.37% Top-1 accuracy on Reading Aloud EEG and 41.43% on Passive Listening EEG out of 101 candidates. Ablation studies show that the two alignment axes are highly complementary: combining them yields consistently better performance than either alone. To the best of our knowledge, this is the first study on decoding large-vocabulary Chinese sentences from non-invasive EEG during overt speech production, and achieving strong classification performance with relatively large closed-set candidate-sentence setting.
