Search papers, labs, and topics across Lattice.
This study addresses the gap in interpretable evaluation metrics for prosody and rhythm in speech-to-speech AI agents by leveraging a large dataset of dyadic English conversations. By constructing matched reference regimes for key prosodic features and implementing a percentile-based evaluation protocol, the authors demonstrate that their method yields more accurate and interpretable assessments compared to traditional pooled human statistics. The results indicate that matched references significantly reduce over-flagging issues and enhance the interpretability of deviations in prosodic features, providing a valuable tool for evaluating conversational AI outputs.
Matched reference regimes for prosody evaluation reveal that traditional methods over-flag deviations, leading to misinterpretations in speech AI assessments.
Speech-to-speech (S2S) AI agents are advancing rapidly, yet evaluation lacks interpretable speech-native measures for conversational prosody and rhythm. Because $F_0$, speaking rate, articulation rate, and pausing shift with model-predicted speaker traits and interaction state, pooled human statistics can be poorly calibrated for evaluating a particular output. Using 4000+ hours of dyadic English conversation from the Seamless Interaction dataset, we construct matched reference regimes for $F_0$ mean, $F_0$ expressivity, speech rate, articulation rate, pause ratio, and mean pause duration. We then define a percentile-based evaluation protocol: extract the same metrics from an S2S output waveform, compare them to the closest matched human reference stratum, and report percentile deviations or 5th-95th percentile out-of-regime flags. On held-out human rows, pooled references over-flag state-conditioned $F_0$ expressivity and rhythm, while matched references return flag rates closer to the nominal 10% and make deviation direction interpretable. These outputs serve as behavioral plausibility checks that complement, rather than replace, perceptual and user-centered evaluation.
Amazon Science