Mengchun Zhang

University of Pittsburgh, Pittsburgh, PA, USA Abstract Cross-subject generalization in EEG-based brain-computer interfaces (BCIs) remains challenging due to individual variability in neural signals. We investigate whether spectral representations offer more stable features for cross-subject transfer than temporal waveforms. Through correlation analyses across three EEG paradigms (SSVEP, P300, and Motor Imagery), we find that spectral features exhibit consistently higher cross-subject similarity than temporal signals. Motivated by this observation, we introduce ASPEN, a hybrid architecture that combines spectral and temporal feature streams via multiplicative fusion, requiring cross-modal agreement for features to propagate. Experiments across six benchmark datasets reveal that ASPEN is able to dynamically achieve the optimal spectral-temporal balance depending on the paradigm. ASPEN achieves the best unseen-subject accuracy on three of six datasets and competitive performance on others, demonstrating that multiplicative multimodal fusion enables effective cross-subject generalization. 1 Introduction Cross-subject generalization remains a fundamental bottleneck in EEG-based brain-computer interfaces (BCIs). Models trained on multi-subject data often degrade substantially when deployed to new users, requiring lengthy subject-specific calibration that undermines the goal of plug-and-play systems (Wan et al., 2021; Liang et al., 2024b). This is due to inherent differences between individuals such as skull thickness, cortical folding, and electrode placement that can produce substantial variation in signal amplitude, timing, and spatial distribution (Lu et al., 2024; Roy et al., 2019). A growing body of work has addressed this limitation through increasingly expressive temporal modeling, progressing from compact CNN-based decoders (Lawhern et al., 2018) to Transformer architectures that capture global dependencies (Song et al., 2022). However, temporal waveforms are highly sensitive to phase shifts, latency jitter, and amplitude scaling across subjects. The hypothesis we investigate here is that spectral representations provide a more stable basis for cross-subject transfer. Frequency-domain features abstract away precise timing information while preserving the oscillatory signatures, such as μ\mu (8–12 Hz) and β\beta (13–30 Hz) rhythms, that serve as primary biomarkers for BCI paradigms (Ang et al., 2008; Mane et al., 2020). To test this hypothesis, we first conduct a systematic correlation analyses comparing temporal and spectral representations across SSVEP, P300, and Motor Imagery paradigms. Our analysis reveals that spectral features exhibit substantially higher cross-subject similarity than temporal signals, suggesting that frequency-domain representations offer a more robust foundation for generalization. Figure 1: ASPEN architecture. Raw EEG is processed through parallel temporal and spectral streams, combined via multiplicative fusion before classification. Motivated by this finding, we introduce ASPEN (Adaptive Spectral Encoder Network, Figure 1), a hybrid framework that processes EEG signals through parallel temporal and spectral streams and combines them via multiplicative fusion. Unlike prior approaches that concatenate or average multimodal features (Li et al., 2021; 2025), multiplicative fusion computes element-wise products of projected stream representations, requiring both streams to agree for a feature to propagate. This cross-modal gating naturally suppresses artifacts and noise that appear prominently in only one view, while amplifying genuine neural patterns that manifest consistently across both temporal and spectral domains. We evaluate ASPEN across six benchmark datasets spanning three paradigms. Our experiments reveal that the optimal spectral-temporal balance varies by task: P300 decoding benefits strongly from spectral emphasis, while Motor Imagery requires greater temporal contribution. ASPEN achieves the best unseen-subject accuracy on three datasets (Lee2019 SSVEP, BNCI2014 P300, and Lee2019 MI), outperforming both specialized temporal models and recent multimodal transformers. These results demonstrate that our model is able to perform cross-subject generalization across different BCI tasks while maintaining robustness across diverse neural signatures. 1.1 Related Work Temporal modeling: Deep learning for EEG signals has evolved from high-capacity architectures like DeepConvNet (Schirrmeister et al., 2017) toward compact, neurophysiologically-informed models. EEGNet (Lawhern et al., 2018) introduced depthwise and separable convolutions that mirror traditional spatial filtering, achieving strong performance with minimal parameters. Transformer-based models such as EEG Conformer (Song et al., 2022) and hybrid CNN-Transformer architectures like CTNet (Zhao et al., 2024) capture long-range temporal dependencies. Temporal convolutional networks (TCNs) offer improved sequential modeling with training stability advantages over recurrent approaches (Ingolfsson et al., 2020; Musallam et al., 2021). Spectral and filter-bank approaches: Filter-bank methods decompose EEG into frequency sub-bands before learning spatial filters. The foundational FBCSP algorithm (Ang et al., 2008) demonstrated that isolating discriminative frequency bands improves motor imagery classification. Deep learning extensions apply this principle with learnable filters (Mane et al., 2020; Liu et al., 2022), while IFNet (Wang et al., 2023) models cross-frequency interactions. Time-frequency representations via wavelets have also shown promise for capturing non-stationary dynamics (Morales and Bowers, 2022). Multimodal fusion: Recent work has begun combining temporal and spectral features. Li et al. (Li et al., 2021) proposed a temporal-spectral squeeze-and-excitation network for motor imagery. TSformer-SA (Li et al., 2025) integrates temporal signals with wavelet spectrograms through cross-view attention for RSVP decoding. Dual-branch architectures have also been explored for emotion recognition (Luo et al., 2023). However, these approaches typically employ additive fusion strategies like concatenation, averaging, or learned weighted sums that allow each stream to contribute independently. Our multiplicative approach instead enforces cross-modal agreement, acting as a feature-wise AND gate that filters unreliable activations. Cross-subject generalization: Domain adaptation techniques including adversarial alignment (Ganin et al., 2016), distribution matching, and adaptive transfer learning (Zhang et al., 2021) have been studied to reduce calibration requirements. MultiDiffNet (Zhang et al., 2025) takes a different route, learning a shared latent space jointly optimized via diffusion, contrastive, and reconstruction objectives to improve cross-subject generalization without explicit distribution alignment. 2 Methodology 2.1 Datasets We evaluated our framework across 3 EEG paradigms (SSVEP, P300, and Motor Imagery) using 6 benchmark datasets: Wang2016 (Wang et al., 2016) and Lee2019 (Lee et al., 2019) for SSVEP; BI2014b (Korczowski et al., 2019) and BNCI2014_009 (Aricò et al., 2014) for P300; and BNCI2014_001 (Tangermann et al., 2012) and Lee2019 (Lee et al., 2019) for MI. Data were partitioned into training, validation, and two distinct test sets: a seen-subject (cross-session) split and an unseen-subject (cross-subject) split to evaluate generalization. Dataset specifications are summarized in Appendix A. 2.2 Domain Characterization Figure 2: Cross-session (left) and cross-subject (right) correlation comparison between temporal and spectral representations. Spectral features exhibit consistently higher cross-subject similarity across all datasets. To inform our architectural choices, we carried out systematic correlation analyses in both the temporal and spectral domains across all EEG datasets. For each dataset, we constructed class-conditional representative patterns by averaging all trials sharing the same label, producing a time-domain representation of shape (C,T)(C,T). For the spectral analysis, we applied an STFT to each trial (using task-specific parameters from the corresponding dataset configuration), took the power representation |Z​(f,t)|2|Z(f,t)|^{2}, and averaged within each class to obtain a spectral representative pattern of shape (C,F,T′)(C,F,T^{\prime}). To quantify the similarity between any two representative patterns, we flattened and z-normalized each one, then computed their Pearson correlation coefficient. We examined two complementary notions of signal consistency: cross-session and cross-subject correlation. Cross-session correlation measures the agreement between representative patterns derived from different recording sessions of the same subject, aggregated across all session pairs and labels. Cross-subject correlation measures the agreement between representative patterns from different subjects, again aggregated across all subject pairs and labels. As shown in Figure 2, spectral representative patterns consistently exhibited higher cross-subject similarity than their raw temporal counterparts across datasets. This suggests that time-frequency representations capture a more subject-invariant signature of task-relevant neural dynamics, and directly motivates our modeling framework’s emphasis on spectral representations as a robust foundation for cross-subject generalization. 2.3 Signal Processing We process the raw EEG signals into two modalities to accommodate the dual-stream architecture of ASPEN. Temporal Modality: To ensure signal stability and cross-subject consistency, raw EEG data are preprocessed using task-specific bandpass filtering and trial-wise Z-score normalization to achieve zero mean and unit variance. Frequency ranges are tailored to each paradigm: 6-90 Hz for SSVEP to capture high-frequency harmonics (Wang et al., 2016), 1-24 Hz for P300 to isolate the evoked response, and 4-40 Hz for Motor Imagery (MI) to focus on μ\mu and β\beta rhythms (Lee et al., 2019). Signals are downsampled to optimize computational efficiency (see Appendix A for details). Spectral Modality: To obtain frequency-domain representations, we apply the Short-Time Fourier Transform (STFT) to each preprocessed trial on a per-channel basis. For each paradigm, we fix a task-specific set of STFT parameters (fs,nperseg,noverlap,nfft)(f_{s},n_{\mathrm{perseg}},n_{\mathrm{overlap}},n_{\mathrm{fft}}) drawn from the dataset configuration. These parameters are validated through a 27-way ablation study with varying window length, overlap ratio, and Fast Fourier Transform (FFT) size, ensuring that all subjects share a consistent frequency×time\mathrm{frequency}\times\mathrm{time} grid at the input of the spectral stream. Concretely, we compute spectrograms with a Hann window and convert them to power spectra by taking the squared magnitude |Zc​(f,t)|2|Z_{c}(f,t)|^{2} for each channel cc, yielding a