Search papers, labs, and topics across Lattice.
This study introduces GSM-Plus-BN, a novel perturbated benchmark dataset for evaluating mathematical reasoning in Bengali, addressing the lack of resources in this linguistically diverse region. By assessing six open-source large language models (LLMs) on 9,000 evaluation samples, the research reveals that while GPT-OSS-20B excels with a 96.08% accuracy on seed questions, larger models like Llama-3.3-70B and GPT-OSS-120B show greater robustness against perturbations. The findings highlight a significant performance gap compared to English benchmarks, emphasizing the challenges posed by perturbed Bengali text and the need for further exploration in this area.
Despite high accuracy in seed questions, LLMs struggle with perturbations in Bengali, revealing a critical gap in mathematical reasoning capabilities compared to English.
The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English. This has created a significant barrier to the equitable development and deployment of AI in linguistically diverse regions such as Bangladesh, where over 230 million people speak Bengali. Despite this global significance, there has been minimal prior work on mathematical reasoning in Bengali and no existing research that systematically benchmarks a perturbated Bengali mathematical dataset, leaving a critical void in assessing model robustness and true comprehension beyond pattern recognition. This study addresses this gap by introducing GSM-Plus-BN, a novel perturbated Bengali mathematical dataset derived from the English GSM-Plus benchmark and verified by human translators. We evaluate six open-source LLMs Qwen3-32B, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct, GPT-OSS-120B, and GPT-OSS-20B using a benchmark of 9,000 evaluation samples comprising 1,000 seed questions and 8,000 perturbed variants under both Standard Prompting and Chain-of-Thought (CoT) Prompting. Experimental results show that GPT-OSS-20B achieves the highest seed question accuracy of 96.08% under Standard Prompting, while larger models such as Llama-3.3-70B and GPT-OSS-120B demonstrate superior robustness across perturbation types. Furthermore, CoT prompting substantially improves reasoning for most models compared to Standard Prompting, yet a notable performance gap persists across all models relative to their English benchmarks, underscoring the inherent difficulty of perturbed Bengali text. This research makes a foundational contribution by providing GSM-PLUS-BN as a new resource and baseline for future Bengali mathematical reasoning research.
