Search papers, labs, and topics across Lattice.
This paper introduces UniSAE, a unified framework for speech attribute editing that allows for simultaneous modifications of speaker, emotion, and content at varying granularities, from sub-phoneme to word level. By leveraging a Discrete Phonetic PosteriorGram (DPPG) representation, the framework effectively encodes phoneme identity, pronunciation variants, and duration, facilitating precise editing capabilities. Experimental results highlight UniSAE's ability to achieve accurate control over speaker and emotion attributes while enabling flexible content editing, all within a single architecture.
UniSAE enables seamless, granular editing of speech attributes, allowing for precise control over speaker, emotion, and content in a unified framework.
Speech editing aims to modify specific portions of an utterance while preserving the remaining speech. Existing approaches primarily focus on word-level content modification and typically treat content, speaker, and emotion editing as separate tasks, limiting both editing granularity and flexibility. We propose UniSAE, a unified speech attribute editing framework which supports composable speaker, emotion and content editing from sub-phoneme to word level within a single architecture. UniSAE introduces a Discrete Phonetic PosteriorGram (DPPG) representation that factorizes speech content into discrete tokens encoding phoneme identity, pronunciation variants, and duration, enabling direct phoneme- and sub-phoneme-level editing. For higher-level modifications, an autoregressive content transformer predicts edited DPPG sequences for word-level content editing. The edited sequences are rendered into speech by a diffusion-based acoustic decoder, conditioned on disentangled speaker and emotion representations. Experimental results demonstrate that the proposed unified framework supports precise speaker and emotion control, content editing at multiple granularities, and joint modification of all three attributes within a single framework.
