Search papers, labs, and topics across Lattice.
The paper introduces IndicEval, a bilingual (English and Hindi) benchmark using real Indian high-stakes examination questions (UPSC, JEE, NEET) to evaluate LLMs' reasoning, domain knowledge, and bilingual adaptability. Experiments with Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B demonstrate that Chain-of-Thought prompting improves accuracy, significant cross-model performance disparities exist, and multilingual degradation remains a challenge. IndicEval offers a scalable and extensible platform for evaluating LLMs in multilingual educational settings.
LLMs still struggle with multilingual reasoning, exhibiting marked accuracy drops in Hindi compared to English on real-world Indian educational benchmarks, even with chain-of-thought prompting.
The rapid advancement of large language models (LLMs) necessitates evaluation frameworks that reflect real-world academic rigor and multilingual complexity. This paper introduces IndicEval, a scalable benchmarking platform designed to assess LLM performance using authentic high-stakes examination questions from UPSC, JEE, and NEET across STEM and humanities domains in both English and Hindi. Unlike synthetic benchmarks, IndicEval grounds evaluation in real examination standards, enabling realistic measurement of reasoning, domain knowledge, and bilingual adaptability. The framework automates assessment using Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting strategies and supports modular integration of new models and languages. Experiments conducted on Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B reveal three major findings. First, CoT prompting consistently improves reasoning accuracy, with substantial gains across subjects and languages. Second, significant cross-model performance disparities persist, particularly in high-complexity examinations. Third, multilingual degradation remains a critical challenge, with marked accuracy drops in Hindi compared to English, especially under Zero-Shot conditions. These results highlight persistent gaps in bilingual reasoning and domain transfer. Overall, IndicEval provides a practice-oriented, extensible foundation for rigorous, equitable evaluation of LLMs in multilingual educational settings and offers actionable insights for improving reasoning robustness and language adaptability.
