Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
SteerSpeech introduces a lightweight activation-steering framework for fine-grained, inference-time emotion control in pretrained text-to-speech (TTS) models without requiring costly retraining. By training low-rank transforms for each target emotion and employing a novel two-pass generation-and-replay pipeline with a straight-through estimator, the method injects steering vectors into hidden activations to modulate emotion while preserving speaker identity and linguistic content. Evaluations demonstrate significantly improved continuous emotion control and speaker identity preservation compared to baseline methods.
Fine-grained emotion control in TTS is now achievable at inference time with minimal speaker or content degradation, bypassing the need for expensive model retraining.
Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.