Search papers, labs, and topics across Lattice.
To resolve the lack of standardized evaluation across heterogeneous speech BCIs, the authors derive open-vocabulary mutual information (OVMI), an information-theoretic metric that quantifies transmitted information relative to a user's open-vocabulary reference word distribution. This framework addresses a fundamental blind spot in standard metrics like Word Error Rate (WER), which evaluate only closed, system-supported vocabularies and thus systematically overestimate true communication capability. Applying OVMI exposes previously obscured trade-offs between vocabulary breadth and decoding precision, while optimizing vocabulary selection to maximize OVMI yields up to a 16.3% relative improvement in accuracy across three distinct speech domains.
Standard word error rates artificially inflate speech BCI performance by ignoring unmodeled language, masking a critical capability trade-off that an open-vocabulary information-theoretic metric resolves while improving decoding accuracy by up to 16.3%.
Speech brain-computer interfaces (speech BCIs) translate neural activity into language, offering a path towards restoring speech for people with paralysis and, more broadly, enabling new forms of natural human-computer interaction. Despite this promise, the field lacks a common measure of progress because systems use different datasets, recording methods, types of speech, and vocabularies, so their reported scores are rarely comparable. Underlying this measurement problem are two unresolved questions: (i) what distribution of words should a speech BCI enable a user to communicate, and (ii) how much information from this distribution can a system convey. We address both by deriving open-vocabulary mutual information (OVMI), an information-theoretic quantity that measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate. This allows capabilities measured under different conditions, such as distinct vocabularies, to be evaluated on a common communication scale. We show that ordinarily reported accuracy, word error rate (WER), and other metrics computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate. We then use OVMI to compare existing systems, expose trade-offs between how much of the user's language a system supports and how accurately it decodes those words, show that these comparisons depend on what the user is expected to communicate, and demonstrate that selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. OVMI therefore provides the speech BCI community with a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field.
