Search papers, labs, and topics across Lattice.
This study investigates the alignment between speaker embeddings generated by speech foundation models and human perceptions of speaker similarity, utilizing over 40 models to conduct a comparative analysis. By measuring the numerical distances between model-derived embeddings against human similarity judgments, the research reveals significant discrepancies in how models represent speaker similarity compared to human listeners. The findings highlight key model configuration factors that can enhance the perceptual accuracy of speaker embeddings, paving the way for more human-like speech processing systems.
Speaker embeddings from foundation models often misalign with human perceptions of similarity, revealing critical gaps in current AI speech processing.
This study presents a comparative analysis between the speaker embeddings of speech foundation models and human subjective perception of speaker similarity. Human listeners have the ability to judge speaker similarity on a continuous scale discerning how similar two voices are. In contrast, speech foundation models embed speaker characteristics into numerical representation. However, a question remains: does the numerical distance between speaker embeddings in these models truly align with the similarity perceived by humans? To address this, we conduct a comprehensive investigation using more than 40 models to compare model-derived distances with human-perceived similarity scores. Furthermore, we identify which factors in model configuration contribute most to a speaker embedding that mirrors human perception. Our findings provide insights for the development of more perceptually grounded speech foundation models.
