Search papers, labs, and topics across Lattice.
This paper introduces MTEB-PT, a benchmark specifically designed for evaluating Portuguese sentence encoders, addressing the lack of language-specific metrics in text embedding evaluation. The study evaluates 17 embedding models using this benchmark, revealing that performance varies significantly across tasks and that multilingual rankings do not accurately reflect Portuguese model effectiveness. Key findings indicate that models with enhanced long-context capabilities excel in longer-input tasks, and language-specific fine-tuning notably boosts performance, particularly in Semantic Textual Similarity tasks.
Multilingual rankings fail to predict Portuguese sentence encoder performance, revealing the critical need for language-specific benchmarks.
Portuguese remains underrepresented in text embedding evaluation, despite being one of the most widely spoken languages in the world. As a result, embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear. We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB, comprising 14 existing datasets across Semantic Textual Similarity (STS), classification, retrieval, and reranking. We use this benchmark to evaluate 17 open- and closed-source embedding models under a unified protocol. Our results show that Portuguese performance is strongly task-dependent: multilingual rankings do not reliably predict Portuguese-specific performance across task families, no single model dominates all settings, and models with stronger long-context capacity are particularly advantageous on longer-input tasks such as retrieval and reranking. The benchmark also shows that language-specific fine-tuning still improves model performance in Portuguese, especially on task types that match the adaptation data most closely. To examine this effect, we fine-tune three representative backbone models with Portuguese contrastive supervision and Matryoshka Representation Learning (MRL). These benchmark-informed baselines yield their strongest gains on STS, consistent with the predominantly symmetric supervision used during training, while also improving retrieval and remaining competitive under dimensional truncation. We release the MTEB-PT benchmark, the fine-tuned models, and the training and evaluation code.
