Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors developed AraDynFact, an automated evaluation framework that dynamically extracts factual information from Arabic Wikipedia to generate culturally grounded question-answering probes for LLMs. This framework addresses the critical bottleneck of static, expensive human curation in multilingual evaluation while probing regional and historical knowledge that generic reasoning benchmarks miss. Auditing leading Arabic-specialized and general-purpose frontier LLMs reveals that this fully automated, dynamic approach correlates strongly with existing labor-intensive, hand-crafted Arabic benchmarks.
Measuring deep Arabic cultural and factual knowledge no longer requires laborious manual curation: dynamically generated Wikipedia probes track model capabilities just as reliably as expensive hand-crafted benchmarks.
As Large Language Models (LLMs) continue to scale both in size and capabilities, their proficiency in the Arabic Language has seen significant advancement. However, a critical gap remains: the extent of their factual knowledge and cultural sensitivity to the diverse Arabic-speaking world remains largely underexplored. Current evaluation metrics often focus on translation or generic reasoning, failing to capture the rich historical, social, and regional nuances inherent to Arabic culture. In addition, most benchmarks rely on heavy work, with human intervention in some steps, making the evaluation of knowledge coverage expensive and slow. To address this deficiency, we introduce AraDynFact, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs. Unlike static benchmarks, AraDynFact employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way. We apply AraDynFact to Arabic Wikipedia and audit the performance of several state-of-the-art models, ranging from Arabic-centric specialized LLMs to high-resource general purpose LLMs. In addition we found a high degree of correlation with existing, hand-crafted Arabic-centric benchmarks, confirming the potential of our dynamic approach.