Search papers, labs, and topics across Lattice.
This paper introduces PluraMath, a new dataset that extends the evaluation of mathematical reasoning in Large Language Models (LLMs) to 18 underrepresented languages across six language families. By benchmarking 27 reasoning LLMs of varying sizes, the study reveals a significant performance gap in mathematical reasoning capabilities between high-resource and underrepresented languages, primarily linked to instruction-following abilities. The open-sourcing of the dataset and evaluation framework aims to democratize access to multilingual benchmarks and foster further research in this area.
LLMs exhibit a stark performance disparity in mathematical reasoning, with underrepresented languages lagging significantly behind their high-resource counterparts.
Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.
