Search papers, labs, and topics across Lattice.
This study introduces the Pancasila-Dilemmas dataset, consisting of 1,834 questions based on Indonesian values, specifically designed to evaluate the value alignment of large language models (LLMs) in the context of Indonesia. By assessing 50 LLMs against this dataset, the authors found that all models scored below 0.5 in Probability Match Score and 0.72 in Max-Vote Agreement Score, indicating a substantial misalignment with Indonesian values, particularly in areas of Religion and Unity. The findings underscore the need for more culturally relevant evaluations of LLMs to ensure their responses align with local human values and intentions.
LLMs struggle significantly with Indonesian values, scoring below 0.5 in alignment, particularly in dilemmas related to Religion and Unity.
The value alignment of large language models (LLMs) is crucial for ensuring responses align with human intention and value preferences. However, most evaluations of value alignment focus on Western or universal values, while assessments grounded in the value systems of specific countries remain scarce. In this paper, we introduce Pancasila-Dilemmas, an evaluation dataset of 1,834 questions derived from Indonesian news, classified by 5 values of Pancasila: Religion, Humanity, Unity, Democracy, and Social Justice. This dataset reflects daily life in Indonesia, making it suitable for measuring the value alignment of LLMs deployed for Indonesia. To ensure a more rigorous evaluation, we choose scenarios containing dilemmas. The dataset is proofread by native speakers and answered by 5 diverse Indonesian citizens. We evaluate 50 closed- and open-source LLMs on our dataset. Results reveal that all evaluated LLMs achieves less than 0.5 Probability Match Score (PMS) and 0.72 Max-Vote Agreement Score (MVAS). Compared by each values, LLMs mostly struggle in Religion and Unity dilemma cases. This highlights a significant gap in capturing Indonesian values. The dataset is publicly available at https://github.com/tjunlp-lab/Pancasila-Dilemmas.
