Search papers, labs, and topics across Lattice.
KinyaEmbed is the first dedicated sentence embedding model for Kinyarwanda, addressing the limitations of existing multilingual models that underperform due to the language's under-representation in pre-training datasets. The model employs a four-stage curriculum training approach using MultipleNegativesRankingLoss, leveraging a diverse range of data sources including paraphrase pairs, entailment structures, and translation pairs to enhance its performance. Evaluated against new benchmarks, KinyaEmbed outperforms leading multilingual models by significant margins, achieving a Spearman correlation of 0.7298 on SemRel2024-rw and the highest document clustering silhouette score among all tested models.
KinyaEmbed outperforms state-of-the-art multilingual models by over 40% in Kinyarwanda sentence embeddings, setting a new standard for low-resource languages.
We present KinyaEmbed, the first dedicated sentence embedding model for Kinyarwanda, a morphologically rich Bantu language spoken by over 12 million people in Rwanda. Existing multilingual embedding models such as LaBSE, mE5-large, and OpenAI text-embedding-3-large perform poorly on Kinyarwanda due to severe under-representation in their pre-training corpora. KinyaEmbed is built on KinyaBERT-large and trained via a four-stage curriculum using MultipleNegativesRankingLoss (MNRL): Stage 1 leverages ~18,000 paraphrase pairs from the Official Gazette of Rwanda with three temperature scales; Stage 2 fine-tunes on 715 NLLB-translated MNLI triplets for entailment structure; Stage 3 aligns representations using English-Kinyarwanda OPUS-100 translation pairs; Stage 4 refines with 2,936 high-quality pairs filtered from KinyaCOMET at quality threshold 0.8. We evaluate on SemRel2024-rw and introduce Wiki-RW-STS, a new contamination-free Kinyarwanda STS benchmark of 300 pairs derived from Kinyarwanda Wikipedia. A seven-checkpoint ensemble (all5+23A*2, with the final stage double-weighted) achieves Spearman \r{ho}=0.7298 on SemRel2024-rw, surpassing mE5-large by 20.9% and OpenAI text-embedding-3-large by 41.0%. KinyaEmbed also achieves the best document clustering silhouette score (0.2146) across all evaluated models. All checkpoints, the KinyaCOMET filtered pairs, and the Wiki-RW-STS benchmark are publicly available.
