Search papers, labs, and topics across Lattice.
This paper introduces TokenSE, a Mamba-based speech enhancement framework operating in the discrete token space of neural audio codecs to improve speech intelligibility for cochlear implant users. By leveraging Mamba's linear complexity, TokenSE overcomes the quadratic complexity limitations of Transformer-based approaches for long audio sequences. Objective and subjective evaluations demonstrate that TokenSE outperforms baseline methods and provides a clear benefit in speech intelligibility for CI users in noisy and reverberant environments.
Mamba's linear attention lets cochlear implant users hear speech more clearly in noisy environments, outperforming previous Transformer-based approaches.
Speech enhancement (SE) is critical for improving speech intelligibility and quality in real-world environments, particularly for cochlear implant (CI) users who experience severe degradations in speech understanding under noisy and reverberant conditions. In this study, we propose TokenSE, a discrete token-based SE framework operating in the neural audio codec space, which predicts clean codec token indices from degraded speech using a Mamba-based model. Unlike the earlier Transformer architecture, whose self-attention mechanism has a computational complexity that grows quadratically with sequence length, the input-dependent selection mechanism of Mamba achieves linear complexity, making it a compelling alternative to Transformers, especially for CI and hearing-aid (HA) applications. Objective evaluations show that TokenSE consistently outperforms baseline methods on both in-domain and out-of-domain datasets. Moreover, subjective listening experiments with CI users indicate clear benefit in speech intelligibility under adverse noisy and reverberant environments.
