Search papers, labs, and topics across Lattice.
This paper introduces ProPS, a framework that generates distributions of speaker embeddings conditioned on natural language prompts, enhancing the utility of x-vectors in representing speaker identity. By converting profile descriptions into sentence embeddings and employing a mixture density network, ProPS effectively predicts Gaussian mixture models in the x-vector space, ensuring that generated embeddings align with specified attributes such as age, gender, and accent. The results demonstrate that ProPS not only preserves these attributes in generated x-vectors but also improves controllability in applications like Text-To-Speech and Voice Conversion systems.
ProPS can generate speaker embeddings that accurately reflect complex attributes from simple natural language prompts, revolutionizing how we synthesize speaker identities.
Speaker embeddings, or x-vectors, are widely used to represent speaker identity and speaker-related attributes, but existing embedding extractors are typically descriptive rather than generative: they map an observed speech segment to an x-vector, which is then used for downstream applications. We introduce ProPS, Prompted Profile Synthesis, a framework for generating distributions of speaker embeddings conditioned on natural language prompts such as"a thirties male speaker with an Indian accent". ProPS converts human-written profile descriptions into sentence embeddings and uses a mixture density network trained on a large-scale dataset to predict a Gaussian mixture model in the x-vector space. The model is trained by maximizing the likelihood that real speaker embeddings match the requested profile, and its generated distributions are evaluated by negative log-likelihood on held-out x-vectors and by attribute classification accuracies on sampled synthetic x-vectors. Experiments show that ProPS produces profile-conditioned distributions and generates x-vectors that preserve requested speaker attributes such as age, gender, accent, and prosodic characteristics. This design enables controllable speaker-profile synthesis for speech generation systems like Text-To-Speech (TTS) or Voice Conversion (VC) while anchoring generated distributions in observed speaker-embedding structure.
