Search papers, labs, and topics across Lattice.
This paper introduces a novel approach called self-guidance, which enhances the performance of neural speech codecs by aligning the decoder's internal feature manifolds when processing quantized tokens and their continuous embeddings. By employing a lightweight feature-mapping loss, the method achieves state-of-the-art low-bitrate performance with a 4x reduction in codebook size while maintaining reconstruction fidelity. Extensive experiments validate the generality of this approach across various inductive biases, demonstrating significant improvements in downstream text-to-speech synthesis tasks.
Achieving state-of-the-art low-bitrate performance with a 4x reduction in codebook size, self-guidance revolutionizes neural audio coding without sacrificing fidelity.
Neural speech codecs based on Vector-Quantized VAEs (VQ-VAEs) are core audio tokenizers for speech LLMs, yet their reconstruction fidelity is bottlenecked by quantization error. Modifying the quantizer or increasing model capacity are common fixes, but they complicate downstream language modeling. Our core idea is to align the decoder's internal feature manifolds when processing both the quantized tokens and their original continuous embeddings, using a lightweight feature-mapping loss. This requires minimal training overhead and no inference-time changes. Applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance. Notably, it enables a 4x codebook reduction without fidelity loss, which downstream TTS experiments show significantly improves LLM-based synthesis by simplifying the token modeling space. Multiple statistical observations and visualizations corroborate the enhanced internal manifold alignment in the decoder. Extensive experiments confirm its generality across various inductive biases. Self-guidance thus establishes an efficient, broadly applicable method for high-fidelity neural audio coding.
Tsinghua AI