Search papers, labs, and topics across Lattice.
This paper conducts a benchmark-based comparative assessment of publicly benchmarked Indian foundation models, evaluating them against global counterparts across eight capability domains. The findings reveal that while Indian models perform well on established benchmarks like MMLU and MATH-500, they lag in participation in newer evaluations and exhibit uneven benchmark coverage among organizations. The authors introduce a Benchmark Maturity Index (BMI) to better assess model capabilities and highlight that many perceived capability gaps may stem from evaluation-ecosystem deficiencies rather than actual model performance.
Indian foundation models excel in traditional benchmarks but struggle with newer evaluations, revealing critical gaps in the national AI ecosystem.
Governments increasingly fund indigenous foundation models to strengthen national AI capability, digital sovereignty, and multilingual computing. Assessing the progress of such national ecosystems is complicated by inconsistent benchmark reporting, proprietary evaluation methodologies, and rapidly evolving model releases. This paper presents a structured, benchmark-based comparative assessment of publicly benchmarked Indian foundation models against global frontier and comparable-scale models, across eight capability domains: general-purpose reasoning, coding and software engineering, agentic AI and computer use, cybersecurity, vision and image understanding, video and multimodal understanding, scientific research, and Indic language capability. Using only publicly reported benchmark results, we find that Indian models achieve strong scores on established benchmarks such as MMLU and MATH-500. However, these benchmarks are now widely regarded as saturated, and frontier developers no longer report them. Indian models participate far less frequently in newer, agentic, and domain-specialized evaluations. Benchmark participation is also highly uneven across Indian organizations. Among the models surveyed, Sarvam AI reports the broadest benchmark coverage by a substantial margin. We propose an exploratory four-dimension Benchmark Maturity Index (BMI), scoring each capability domain on standardization, participation, independent verification, and national coverage. We show that the BMI refines, and in some cases revises, the maturity judgments that a purely descriptive review would produce. We argue that many apparent capability gaps in the public record cannot be distinguished, on available evidence, from evaluation-ecosystem gaps. This has direct implications for how national AI programs should design monitoring and funding criteria.
