Search papers, labs, and topics across Lattice.
This paper addresses the issue of stochastic catastrophic failures in autoregressive neural-codec text-to-speech (TTS) models, which can lead to silence, early termination, or hallucinated outputs. By employing a best-of-N ASR self-verification method, the authors achieve near-zero catastrophic failure rates across multiple TTS systems and codecs, demonstrating that failures can be effectively mitigated with minimal computational overhead. Additionally, they introduce a distillation technique that incorporates self-verified behavior into the model, significantly improving robustness during single-shot decoding without incurring test-time costs.
Near-zero catastrophic failures in TTS models can be achieved through simple ASR self-verification, revolutionizing the reliability of speech synthesis.
Open autoregressive neural-codec text-to-speech (TTS) models sound excellent on typical inputs yet suffer stochastic catastrophic failures: on a meaningful fraction of utterances they emit silence, terminate early, or collapse into repetitive or hallucinated content. We show this failure mode is cheap to remove. Under a single format-robust metric (a catastrophic-failure rate via an ASR round-trip), best-of-N ASR self-verification drives failures to near-zero: no observed failures remain by N=2 on a standard corpus (LibriSpeech) and by N=4 on a hard prompt set. This is not an artifact of one model: the reduction replicates across four open codec-TTS systems and three neural codecs (XCodec2, SNAC, Mimi), reaching the near-zero floor by N=2 on three of the four. We then make the fix free at inference time by distilling the self-verified behaviour into the model, which recovers much of the robustness in single-shot decoding, closing ~52-58% of the failure mass on hard inputs at no test-time cost. The distillation gain concentrates where it is needed (hard inputs); on already-reliable prose there is no headroom and no detectable change. A controlled comparison adds a clean negative: offline direct preference optimization (DPO/IPO) does not beat plain supervised distillation, and an online iterative variant is promising but not statistically separable at our evaluation size. We report honestly the one model that resists (a larger Llasa where scale did not obviously help) and a rare-word capability ceiling that no self-distillation method overcomes
