Search papers, labs, and topics across Lattice.
The paper introduces ExCAM, an explainable metric for evaluating cultural awareness in LLM-generated text, addressing the limitations of existing benchmarks that are costly to create and often rely on dated evaluation mechanisms. ExCAM is trained on ExCAM40k, a dataset of instruction-output pairs with synthetic cultural errors derived from nine existing benchmarks. Results show ExCAM achieves up to 80% accuracy in error detection, outperforming baselines including GPT-4, enabling fine-grained and explainable cultural evaluation of free text.
LLMs can now be automatically audited for cultural insensitivity with 80% accuracy, thanks to a new metric that pinpoints and explains cultural errors in generated text.
Evaluating the cultural awareness of large language models is crucial to ensure the fairness of generated text and the generalizability of applications across the world. Recent benchmarks explore cultural goods like food or values like behavior in stressful situations through the lens of question answering or text generation tasks. However, creating these benchmarks requires time-intensive and costly human annotations. Also, benchmarks that evaluate cultural awareness in free text are scarce and often rely on dated evaluation mechanisms. To address this gap, we introduce ExCAM, an Explainable Cultural Awareness Metric, which is, to our knowledge, the first dedicated evaluation metric that identifies, rates and explains cultural errors in instruction-output pairs. To train and evaluate ExCAM, we introduce ExCAM40k, a dataset comprised of nine existing benchmarks that we reformat and enhance with synthetic errors. Compared to several baselines, including GPT-5, ExCAM achieves the highest error detection rate with up to 80% accuracy on a balanced test set. Therefore, ExCAM opens the pathway towards fine-grained and explainable cultural evaluation of free text.
