Search papers, labs, and topics across Lattice.
NRITYAM introduces a benchmark designed to assess the cultural comprehension of language models specifically in the context of global dance traditions. This benchmark consists of 9,260 question-answer pairs across 12 languages, developed through collaboration with native dance artists and speakers to ensure cultural relevance. The evaluation of various models highlights the disparities in their understanding of traditional performing arts, establishing NRITYAM as a critical tool for enhancing AI's socio-cultural awareness.
Language models struggle with cultural nuances, but NRITYAM reveals significant gaps in their understanding of global dance traditions.
Language models have become essential tools in shaping modern workflows. However, their global effectiveness hinges on a nuanced understanding of local socio-cultural contexts. To address this gap, we present NRITYAM, a comprehensive benchmark for evaluating the cultural comprehension capabilities of language models in the context of global dance traditions. NRITYAM comprises 9,260 carefully curated question-answer pairs spanning 12 languages, making it the largest dataset dedicated to evaluating cultural knowledge in dance. The dataset has been developed from the ground up through close collaboration with native dance artists and native speakers of the languages, who authored and validated culturally relevant questions specific to their regions. We evaluate a broad set of models, including large language models, small language models, multimodal large language models, and small multimodal language models. As a multilingual and multicultural benchmark, NRITYAM sets a new standard for evaluating the ability of AI systems to understand and reason about traditional performing arts. Detailed dataset samples are available at~\url{https://github.com/niladrighosh03/NRITYAM}.
