Search papers, labs, and topics across Lattice.
This paper introduces masked autoregressive speech enhancement (MARSE), which utilizes continuous neural audio codec (NAC) representations for iterative decoding of masked clean speech frames. By systematically exploring various decoding policies while maintaining a consistent deep neural network architecture and training setup, the authors demonstrate that MARSE achieves superior speech quality and intelligibility compared to traditional discrete token approaches. The findings reveal a flexible trade-off between enhancement performance and computational efficiency, making MARSE a promising advancement in speech enhancement techniques.
Continuous NAC representations can significantly boost speech quality and intelligibility in enhancement tasks, outperforming traditional discrete methods.
Most previous work on speech enhancement (SE) based on masked generative modeling relied on discrete token representations of audio signals, obtained using neural audio codecs (NACs). However, a recent study has shown that continuous latent representations of NACs can be advantageous for SE in terms of speech quality and intelligibility. In this work, we propose masked autoregressive SE (MARSE), a method for SE based on iterative decoding of masked clean speech frames using continuous NAC representations of speech. In particular, we investigate a set of different decoding policies, ceteris paribus, that is, using the same DNN (a Conformer model), the same NAC (the DAC codec) and the same training setup. The results show that MARSE enables a flexible trade-off between SE performance and computational cost. Audio examples and code are available online.
