Search papers, labs, and topics across Lattice.
This study investigates the representations formed by text-to-speech (TTS) language models, specifically focusing on the CosyVoice3 backbone, by employing BatchTopK sparse autoencoders to analyze the shared residual stream of text and speech tokens. The authors introduce a modality-aware auto-interp pipeline that effectively labels features derived from text prefixes and speech clips, revealing interpretable features such as phonemes, laughter, and speaker attributes. The findings demonstrate that these features can be causally manipulated, allowing for significant control over TTS outputs, such as increasing laughter probability and altering perceived speaker gender while maintaining content integrity.
Causal manipulation of TTS features enables unprecedented control over speech synthesis, transforming how we can steer voice characteristics in real-time.
Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream. We train BatchTopK sparse autoencoders on the LM backbone of CosyVoice3 and introduce a modality-aware auto-interp pipeline that labels each feature from where it fires-text-prefix context, 1-second speech clips, or both. The recovered features are interpretable, spanning phonemes, laughter, accent prompts and speaker gender. Steering through the SAE latent space shows these features are causal rather than merely descriptive: targeted interventions raise laughter probability from 0.02 to 0.79, flip perceived speaker gender, and control speech rate while preserving spoken content. SAE features thus serve both as interpretability objects and as control directions for TTS synthesis.
