Search papers, labs, and topics across Lattice.
This paper introduces the Style Text Embedding Benchmark (STEB), a comprehensive framework designed to standardize the evaluation of style embeddings across various tasks and languages. By analyzing 96 datasets in 7 languages, the study reveals that existing semantic embeddings are inadequate for stylistic tasks and that no single style embedding excels universally across all evaluated tasks. The open-source STEB benchmark aims to unify the fragmented landscape of style embedding evaluation and facilitate future research in this area.
Semantic embeddings falter in stylistic evaluations, revealing a critical gap in current embedding methodologies.
While semantic embeddings are rigorously evaluated on the Massive Text Embedding Benchmark, the evaluation of style embeddings remains fragmented, with each work relying on their own set of tasks and datasets. To bridge this gap, we introduce the Style Text Embedding Benchmark, a comprehensive open-source benchmark intended to standardize the evaluation of style embeddings. STEB encompasses 96 datasets across 7 languages, spanning applications such as authorship verification, authorship retrieval, AI-text detection, probing of linguistic features, and others. We find that semantic embeddings consistently fail in stylistic tasks, and that there is no style embedding that is universally superior across all tasks evaluated. We open-source the STEB code base at: https://github.com/rrivera1849/STEB.
