Search papers, labs, and topics across Lattice.
This paper introduces STEB, a comprehensive benchmark for evaluating speech-to-speech translation (S2ST) that assesses both translation fidelity and expressive attributes such as emotion and nonverbal vocalizations. By employing a caption-then-summarize framework combined with LLM judgment, the authors demonstrate significant correlations with human listener evaluations across multiple expressive dimensions. The evaluation of six S2ST systems reveals a notable gap in expressiveness preservation, particularly in emotional and nonverbal aspects, highlighting a critical challenge in the field.
Expressiveness preservation in speech-to-speech translation remains a significant hurdle, with systems scoring poorly on emotional and nonverbal fidelity despite achieving high translation accuracy.
Speech-to-speech translation (S2ST) should preserve not only lexical meaning, but also expressive attributes: emotion, scenario style (e.g., news reporting vs. dramatic dialogue), and nonverbal vocalizations (NVs). Moreover, collecting cross-lingual target speech that is both translation-faithful and expressively aligned with the source is difficult at scale, making reference-based evaluation impractical. We introduce STEB (Speech-to-Speech Translation Expressiveness Benchmark), a 32.6-hour Chinese--English benchmark that evaluates both standard dimensions (translation fidelity, speaker similarity, duration alignment) and expressiveness dimensions (emotion, scenario style, NV preservation). For expressiveness evaluation, STEB uses a caption-then-summarize framework that converts speech into structured expressive attributes and compares source and hypothesis attributes with an LLM judge. Human validation shows statistically significant correlations with listener judgments across all expressive dimensions. We evaluate six S2ST systems covering cascaded systems, end-to-end models, and speech large language models. Many systems, especially cascaded ones, achieve strong translation fidelity, but they still struggle with emotion preservation (best: 3.82/5) and NV preservation (best: 2.31/5). These results reveal a gap between semantic transfer and expressive transfer, identifying expressiveness preservation as an open challenge for S2ST. Audio samples are available at https://cmots.github.io/steb.github.io/.
Tsinghua AI