Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To overcome weak expressiveness in training-free emotional text-to-speech, the authors decompose internal emotion steering vectors into a shared component that departs from neutrality and a residual component specific to the target emotion. This decomposition enables precise representation steering on frozen backbones (IndexTTS-2 and CosyVoice2) without requiring compute-intensive fine-tuning or scarce labeled datasets. Evaluated on IEMOCAP, their approach improves rank correlation by up to 118.8% relative and emotion hit rate by 20.1% over standard vector steering, with human raters correctly identifying the intended emotion up to 35% more often.
Emotion vectors are not monolithic directions: decoupling the generic shift away from neutrality from the emotion-specific residual more than doubles training-free steering precision in frozen TTS backbones.
Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. We therefore study vector steering, a training-free approach that modifies the internal representations of a frozen model. CoCoEmo, a conventional vector steering method for emotion TTS, treats each emotion vector as an indivisible direction controlled by a single global strength, limiting adherence to the requested emotion. In this work, we first discover that an emotion vector can be decomposed into a shared component that moves speech away from neutral expression and a residual component that directs generation toward the requested emotion. Building on this finding, we propose Emotion Residual-Enhanced Steering for TTS (EmoRES), a novel method that controls the two components without retraining the backbone. On IEMOCAP, EmoRES outperforms CoCoEmo across all four objective emotion metrics on the IndexTTS-2 and CosyVoice2 backbones. Rank correlation improves by 26.13 and 12.97 percentage points, corresponding to relative gains of 118.8% and 33.1%, while emotion hit rate improves by 12.95 and 6.92 points, corresponding to relative gains of 20.1% and 9.8%. Human evaluation further shows a relative improvement up to 35.0% in the rate at which listeners correctly identified the dominant requested emotion and up to a 17.3% improvement in fidelity, while listeners prefer EmoRES for naturalness in up to 63.8% of pairwise comparisons. Component ablations further demonstrate that effective control benefits from preserving the shared component while strengthening the residual of the emotion steering vectors.