Search papers, labs, and topics across Lattice.
This study investigates the mechanisms behind low frame rate degradation in neural audio codecs, particularly focusing on the quality cliff observed at 6.25 Hz. Through a controlled frame rate ablation, the authors find that the degradation is not due to phonemic collisions or codebook saturation, but rather a suboptimal training configuration that limits inter-token context. By correcting the training approach, they demonstrate that word error rates degrade smoothly down to 1.6 Hz, indicating that efficiency gains in low frame rate codecs are more achievable than previously thought.
The quality cliff at low frame rates is driven by training configuration issues, not inherent limitations of neural audio codecs.
Low frame rates in neural audio codecs are attractive for autoregressive speech synthesis, where the generation cost scales linearly with the sequence length. Recent work has demonstrated that codecs can operate at 12.5 Hz and below, but the mechanisms underlying low frame rate degradation remain insufficiently understood. We investigate these mechanisms through a controlled frame rate ablation. We reproduce a quality cliff at 6.25 Hz reported in previous works and evaluate candidate explanations: phonemic collisions and codebook saturation, neither of which shows evidence of a fundamental barrier. The cliff is instead caused by suboptimal training configuration: fixed clip duration during training yields too few tokens at low frame rates, starving the decoder of inter-token context. Once corrected, WER degrades smoothly with phonemic load down to 3.1 Hz and 1.6 Hz, suggesting the inference-time efficiency gains of low frame rate codecs are more accessible than previously assumed.
CMU ML