Search papers, labs, and topics across Lattice.
This study critiques the Best-of-$N$ (BoN) inference method in zero-shot text-to-speech (TTS) by revealing that the effectiveness of an automatic speech recognition (ASR) verifier is significantly influenced by the ASR family used for evaluation. Through experiments on the LibriSpeech-PC dataset, it was found that verifier rankings can reverse depending on the ASR family, with same-family pairs achieving 2-3 times more oracle headroom than cross-family pairs, indicating a deeper issue of evaluator bias rather than mere representational similarity. The authors introduce cross-family rank ensembles that improve word error rates (WER) significantly, achieving a mean WER of 1.61% at N=10, thus advocating for cross-evaluator triangulation as a standard practice in TTS evaluation.
Evaluator bias in TTS assessment can lead to misleading quality rankings, with same-family ASR pairs outperforming cross-family pairs by up to 3x.
Best-of-$N$ (BoN) inference improves content consistency in zero-shot text-to-speech by selecting from $N$ candidates with an automatic speech recognition (ASR) verifier. We identify an underexplored evaluation confound: a verifier's apparent quality depends strongly on which ASR family judges it. On LibriSpeech-PC test-clean~\citep{librispeechpc} with F5-TTS~\citep{f5tts}, verifier rankings reverse across Whisper, wav2vec~2.0, and HuBERT evaluators, and same-family verifier-evaluator pairs recover 2-3$\times$ more oracle headroom than cross-family pairs despite near-identical representations (linear CKA $0.978$) -- a pattern consistent with identity- or lineage-level coupling rather than representational overlap. We propose two \textbf{cross-family rank ensembles} (rank-averaging and conjunctive max-rank) that attain the lowest mean WER across three independent evaluators -- $1.61\%$ at $N{=}10$ ($-12\%$ relative to F5-TTS) -- with no measurable degradation under automatic SIM-o/UTMOS metrics; the best single verifier drives WER from $2.06\%$ to $1.72\%$ ($-16.5\%$) under the official F5-TTS evaluator. We recommend cross-evaluator triangulation as default reporting practice.
