Search papers, labs, and topics across Lattice.
ParaSpeechCLAP is introduced as a dual-encoder contrastive model mapping speech and text style captions into a shared embedding space, handling both intrinsic (speaker-level) and situational (utterance-level) style descriptors. Specialization via ParaSpeechCLAP-Intrinsic and ParaSpeechCLAP-Situational models improves performance on individual style dimensions, while a unified model excels in compositional evaluation. The model's effectiveness is demonstrated through style caption retrieval, speech attribute classification, and as a reward model for style-prompted TTS, outperforming baselines across these tasks.
Style-controllable speech synthesis just got a major upgrade: ParaSpeechCLAP lets you dial in nuanced speaker traits and situational contexts far beyond what existing models can handle.
We introduce ParaSpeechCLAP, a dual-encoder contrastive model that maps speech and text style captions into a common embedding space, supporting a wide range of intrinsic (speaker-level) and situational (utterance-level) descriptors (such as pitch, texture and emotion) far beyond the narrow set handled by existing models. We train specialized ParaSpeechCLAP-Intrinsic and ParaSpeechCLAP-Situational models alongside a unified ParaSpeechCLAP-Combined model, finding that specialization yields stronger performance on individual style dimensions while the unified model excels on compositional evaluation. We further show that ParaSpeechCLAP-Intrinsic benefits from an additional classification loss and class-balanced training. We demonstrate our models'performance on style caption retrieval, speech attribute classification and as an inference-time reward model that improves style-prompted TTS without additional training. ParaSpeechCLAP outperforms baselines on most metrics across all three applications. Our models and code are released at https://github.com/ajd12342/paraspeechclap .
