Search papers, labs, and topics across Lattice.
This paper revisits classifier-free guidance (CFG) in flow-matching-based zero-shot text-to-speech (TTS) by independently masking text and speech-prompt conditions to decompose the guidance field into text, speaker, and joint residuals. The authors identify that conventional speaker-selective guidance can entangle speaker and joint residuals, negatively impacting text generation quality. By introducing joint residual reweighting, they achieve improved speaker similarity without sacrificing text correctness, as validated through experiments on F5-TTS and CosyVoice2 datasets.
Joint residual reweighting allows for enhanced speaker fidelity in zero-shot TTS while maintaining text accuracy, challenging the conventional trade-offs in guidance methods.
Classifier-free guidance (CFG) is widely used in flow-matching-based zero-shot text-to-speech (TTS), where generation is typically controlled by two conditions: the target text and a prompt speech signal. Standard CFG strengthens these conditions jointly, while recent branch-selective guidance methods attempt to enhance text or speaker conditioning separately, often leading to a trade-off between text correctness and speaker similarity. In this paper, we revisit the CFG under independently masked text and speech-prompt conditions, and decompose the guidance field into text, speaker, and joint residuals. We show that conventional speaker-selective guidance entangles the speaker residual with the joint residual, which may disturb text-related generation. Based on this observation, we propose joint residual reweighting, which independently controls the speaker and joint residuals within the standard CFG framework. Experiments on F5-TTS and CosyVoice2 show that the proposed method improves speaker similarity while maintaining competitive text correctness, demonstrating the usefulness of the joint residual for balancing speaker fidelity and text accuracy in zero-shot TTS.
