Search papers, labs, and topics across Lattice.
This paper introduces a unified moral-value dataset specifically designed for instruction tuning of large language models (LLMs), addressing the challenge of aligning these models with human values. By merging existing moral-value datasets and converting them into an instruction-response format, the authors demonstrate that incorporating this dataset into training maintains general-task performance while enhancing value-oriented task performance. The findings suggest that the mixing ratio of general task datasets with the moral-value dataset significantly influences the effectiveness of value alignment in LLMs.
Training LLMs with a unified moral-value dataset can enhance their alignment with human values without sacrificing general performance.
Large language models (LLMs) have developed rapidly and become valuable tools in everyday life. However, how to align LLMs to a particular set of human values is still an open problem. Recent studies show that instruction tuning has strong potential for zero-shot tasks and may serve as an effective approach to addressing value alignment. Nevertheless, although many datasets for instruction tuning already exist, they are not specifically designed around moral scenarios and behaviors. We construct a unified moral-value dataset that can be directly used for instruction tuning. This dataset is built upon existing moral-value datasets by merging them into a unified corpus and converting them into an instruction-response format. We show that training on a mixed dataset combining general task datasets with our dataset preserves general-task performance, and we report preliminary observations on how the mixing ratio affects value-oriented task performance. Our work provides a moral-value dataset for instruction tuning and offers a useful resource for further alignment research. The dataset is available at https://huggingface.co/datasets/teohzzh/value-for-instruction-tuning.
