Search papers, labs, and topics across Lattice.
This study introduces the Cross-Subject Perceived Speech Decoding (CPSD) framework to enhance the decoding of perceived speech from non-invasive brain recordings across different subjects. By employing a two-stage training process that includes contrastive learning for shared representation and personal specialization for target subject adaptation, the framework addresses the challenges of generalizability and high training costs. The CPSD framework significantly outperforms existing methods, achieving improvements in Top-10 accuracy of over 6.8%, 15.4%, and 15.8% across three diverse datasets, underscoring its robustness and efficiency.
Achieving over 15% improvement in speech decoding accuracy across subjects reveals a breakthrough in the generalizability of brain-computer interfaces.
Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechanisms for extracting subject-consistent information. These limitations result in high training costs and suboptimal decoding performance. To address these challenges, we propose an innovative Cross-Subject Perceived Speech Decoding (CPSD) framework, which comprises two training stages: source model pre-training and personal specialization. In the source model pre-training stage, contrastive learning is employed to capture shared representations across multiple source subjects. Subsequently, personal specialization initializes the model for the target subject by extracting consistent components from the source model and fine-tuning it using target subject data. Additionally, we introduce the Positional Encoding-based Spatial Attention (PESA) module, which remaps MEG/EEG data into a standardized reference space, thereby enhancing cross-subject consistency and facilitating model training. We evaluate the proposed CPSD framework on three perceived speech neural datasets encompassing different modalities and languages. The results demonstrate that our framework outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top-10 accuracy on the Armeni 2022, PKUEEG 2025, and Broderick 2018 datasets, respectively. Further analyses confirm the effectiveness, efficiency, and robustness of the proposed approach.
