Search papers, labs, and topics across Lattice.
This paper introduces ALEE, a novel framework for evaluating text embeddings across multiple languages by leveraging English-centric minimal pairs generated through Abstract Meaning Representations (AMR). By addressing the limitations of existing benchmarks, ALEE facilitates targeted diagnostics for semantic similarity tasks in over 275 languages, revealing significant performance disparities based on language prevalence and text characteristics. The findings underscore the critical need for improved cross-lingual semantic representation, particularly for low-resource languages, as performance varies widely across different linguistic contexts.
ALEE uncovers substantial performance gaps in cross-lingual semantic representation that reflect the biases of training data and tokenization strategies across 275+ languages.
Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith (Li et al., 2025) to the cross-lingual and paragraph level. ALEE uses Abstract Meaning Representations (AMR) to generate English minimal pairs with controlled, fine-grained semantic shifts, which are paired with translations in target languages. This approach enables targeted diagnostics for models in any language with English parallel data. We conduct a large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization. We release ALEE at https://github.com/Andrian0s/any-lang-embed-eval
