Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
FMelCodec, a neural speech codec operating at ultra-low bitrates (250-750 bps), is introduced using a three-stage coding-refinement-reconstruction framework. The coding stage uses a highly compressed VQ encoder-decoder with online clustering to maintain codebook diversity, the refinement stage employs conditional flow matching for mel-spectrogram enhancement, and the reconstruction stage uses a HiFi-GAN vocoder. Experiments show FMelCodec achieves higher speech reconstruction quality and speaker similarity than existing methods at these bitrates, with lower computational complexity.
Achieve surprisingly high-quality and speaker-consistent speech reconstruction at ultra-low bitrates (250-750 bps) with a novel neural codec architecture.
Ultra-low-bitrate speech coding is pivotal for bandwidth-constrained communication and deep compression, yet maintaining naturalness and speaker identity at such extreme bit budgets remains challenging due to pronounced information loss and quantization instability. To this end, we propose FMelCodec, an ultra-low-bitrate neural speech codec in the mel-spectrogram domain, cast as a three-stage coding-refinement-reconstruction (CRR) framework that can operate at as low as 250 bps. In the CRR framework, the front-end mel-spectrogram coding stage employs a highly aggressive 640x compression/decompression encoder-decoder structure with a single 1024-entry VQ codebook, coupled with an online clustering strategy that reassigns underused codewords to prevent codebook collapse and preserve codebook diversity. The subsequent conditional flow matching (CFM)-based mel-spectrogram refinement stage leverages a lightweight velocity-field estimator and CFM-based solver to refine the codec-degraded mel-spectrogram produced by the preceding decoder, and adopts a self-consistency training scheme that supports fewer iterative inference steps for the purpose of reducing computational overhead. Finally, the vocoding-driven waveform reconstruction stage employs a HiFi-GAN vocoder to faithfully reconstruct waveform from the refined mel-spectrogram. Experiments conducted on two datasets spanning two sampling rates show that, under ultra-low-bitrate constraints of 250 bps for 16 kHz and 750 bps for 48 kHz, both objective and subjective evaluations consistently demonstrate that FMelCodec achieves higher speech reconstruction quality and speaker similarity, while incurring lower computational and model complexity.