Search papers, labs, and topics across Lattice.
This paper introduces MTEB-PT, a dedicated benchmark for evaluating text embeddings specifically for Brazilian Portuguese, addressing the limitations of existing evaluations that rely on translated corpora. The benchmark encompasses 22 native tasks across seven categories and evaluates 93 models, revealing that an openly licensed model can achieve top-tier performance without the need for commercial APIs. Notably, the study finds that rankings on global multilingual leaderboards do not accurately reflect performance on the native Portuguese tasks, highlighting the importance of localized benchmarks.
An openly licensed model outperforms commercial APIs in Brazilian Portuguese text embeddings, challenging the notion that only proprietary solutions can deliver high-quality results.
Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated. We introduce MTEB-PT, a benchmark of 22 native Brazilian-Portuguese tasks across seven categories (classification, multilabel classification, pair classification, semantic textual similarity, clustering, retrieval, and reranking), admitting only data created or found in Portuguese and excluding translations by construction. We evaluate 93 models spanning 23M to 27B parameters: 73 open-weight and 20 closed commercial APIs. Alongside the leaderboard we report a statistical layer for every headline comparison: per-task bootstrap confidence intervals, paired-bootstrap significance, a task- and instance-level discrimination analysis (how sharply each task separates models) adapted from Item Response Theory, and a cross-leaderboard correlation. Three findings stand out. The benchmark cleanly separates about a dozen tiers of models, though the top six are statistically too close to order. An openly licensed, self-hostable model reaches that leading tier, so strong Portuguese embedding quality does not require a commercial API. And a model's rank on the global multilingual leaderboard predicts its Portuguese rank only moderately (Spearman rho = 0.75 over 55 shared models; one model ranks 3rd there and 49th here), so a native benchmark measures something the multilingual boards do not. We release every task, our code, and a public leaderboard, so practitioners can choose Portuguese embedding models on native evidence.
