Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that uses LLM-driven stochastic transformations to assess embedding model robustness across lexical/stylistic variations, length, and language. Evaluating 16 open-weight embedding models across 32 datasets and 42 languages, they reveal that models exhibit distinct robustness profiles across these axes, and that scaling model size doesn't necessarily close the gap between original and transformed evaluations. They also find that English datasets are more sensitive to HTEB transformations than multilingual datasets, highlighting the limitations of static, one-dimensional robustness metrics.
Current embedding benchmarks hide critical failure modes: models exhibit surprisingly specific and decoupled robustness profiles when exposed to realistic variations in text style, length, and language.
Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embedding robustness is multidimensional, since models respond differently to different types of variation, and requires dynamic evaluation to expose failures hidden by static benchmarks. We introduce the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that challenges model robustness along three practically interpretable axes (Lexical/Stylistic, Length and Language) by stochastically transforming inputs at evaluation time with an LLM. Evaluating 16 open-weight embedding models on 32 datasets covering 42 languages under transformations validated by 4,800 human ratings on an English subsample, we find three patterns: (1) Models exhibit specific, partly decoupled robustness profiles across axes. (2) Across three model families, scale increases absolute scores but does not close the gap between original and transformed evaluations. Here, scaling tends to improve specifically the Language axis. (3) English datasets are more sensitive to HTEB transformations than multilingual datasets. This demonstrates that HTEB identifies strengths and weaknesses of models along deployment-relevant axes, challenging current embedding benchmarks and arguing for multidimensional, dynamic robustness evaluation.