Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel training algorithm, Steerable Cultural Preference Optimization (SCPO), aimed at developing reward models that accurately reflect the diverse preferences of cultural sub-communities without excessive bias. By evaluating its performance across two datasets, PRISM and GlobalOpinionQA, and seven countries, SCPO achieves up to a 7-point improvement in the minority reward model and demonstrates a remarkable 280% increase in training data efficiency compared to traditional full-data finetuning. Additionally, the method effectively mitigates bias through a targeted weighting approach, ensuring fair representation of various cultural perspectives.
Reward models can now be trained to reflect diverse cultural preferences with unprecedented efficiency and reduced bias, enhancing LLM alignment across global communities.
It is essential for large language model (LLM) technology to serve many different cultural sub-communities in a manner that is acceptable to each community. However, research on LLM alignment has so far predominantly focused on predicting a unified response preference of annotators from certain regions. This paper aims to advance the development of alignment models with a more global outlook, that are able to accurately represent the preferences of subcommunities and do not exhibit excessive bias towards any of them. We focus on the development of reward models for this purpose and present a novel reward model training algorithm (SCPO) that can incorporate diverse cultural preferences in a balanced manner. Our method results in performance increases of the minority reward model of up to 7 points over the baseline model across two datasets, PRISM and GlobalOpinionQA, and across 7 countries. SCPO is up to 280% more training data-efficient than full-data finetuning of reward models. In addition, we perform analysis of bias by separately evaluating on the preference of subcommunities and show that excessive bias is mitigated via our weighting method. Our code is available at https://github.com/minsik-ai/Steerable-Cultural-Preference