Search papers, labs, and topics across Lattice.
This study investigates the relationship between spelling rate accuracy and various performance metrics in Event-Related Potential (ERP)-based Brain-Computer Interfaces (BCIs). The authors emphasize the importance of the spelling rate as a key indicator of information transfer rate, particularly in the context of imbalanced data distributions common in ERP applications. Their analysis of two datasets reveals that metrics such as the Brier score, Matthews Correlation Coefficient, and those addressing class imbalance (e.g., ROC AUC, PR AUC) are most indicative of user spelling performance, advocating for their use in future ERP BCI research.
The Brier score and Matthews Correlation Coefficient emerge as top metrics for accurately reflecting spelling performance in ERP-based BCIs, challenging traditional reliance on loss and accuracy.
For predictive models, the often-reported performance metrics are the loss and accuracy. In synchronous Brain- Computer Interface (BCI) systems, these metrics are informative for most BCI paradigms; however, for Event-Related Potential (ERP) applications the spelling rate, which measures the number of characters correctly selected is more important as it influences the estimation of information transfer rate (ITR) and any related metric measuring spelling performance. Moreover, ERP-based BCIs hold imbalanced data class distributions, which require reporting metrics that can handle the imbalance, such as the area under the receiver operating characteristic curve (ROC AUC). In this work, we study the correlation of the spelling rate with 13 metrics to identify which among them best reflect user spelling performance and how they are affected by trial repetition. The Results of two datasets (a private LARESI ERP dataset and the public OpenBMI ERP dataset) favor the Brier score, Matthews Correlation Coefficient (MCC), and the metrics that account for class imbalance in binary classification: ROC AUC, area under the Precision-Recall curve (PR AUC), Average Precision (AP), and partial AUC (pAUC). These findings encourage researchers and practitioners to report those metrics in ERP-based BCI experiments.
