Search papers, labs, and topics across Lattice.
This paper reevaluates the moral reasoning capabilities of large language models (LLMs) using the MoReBench dataset, which contains human-authored rubrics for assessing moral reasoning across 1,000 cases. By allowing LLMs to generate their own scoring rubrics for moral analysis rather than directly scoring their responses, the authors find that the generated rubrics align more closely with human evaluations and reveal the complexity of moral dilemmas. This approach leads to a more optimistic assessment of LLMs' moral reasoning abilities, suggesting they are more competent than previously thought.
LLMs can produce scoring rubrics for moral dilemmas that align better with human evaluations, challenging the notion that they are poor at moral reasoning.
For highly capable AI systems to operate safely in dynamic, open-ended environments, they must be able to identify, understand, and respond to moral reasons for action, and constrain their behaviour accordingly. A growing body of research aims to evaluate this capacity -- moral competence -- in today's most capable AI systems, recently reaching broadly pessimistic conclusions. One of the most ambitious such papers collects gold-standard human-authored rubrics for evaluating moral reasoning in 1,000 cases, and benchmarks frontier AI models against those rubrics, with underwhelming results. In this paper, we argue that the MoReBench dataset can be redeployed to give a much more optimistic picture of LLMs'moral reasoning (an essential part of moral competence). We show that if, instead of scoring LLMs'responses to these cases against these rubrics, we instead give the LLMs the same task given to humans -- to generate scoring rubrics for the moral analysis of particular cases -- the rubrics they generate are both better calibrated to the human rubrics than their open-ended responses, and, where they differ, plausibly reflect nothing more than the vast dimensionality of most moral problems, as well as highlighting some human departures from the"rubric for creating rubrics". Taking these points into consideration, the MoReBench dataset suggests that LLMs are significantly more capable at moral reasoning than was previously believed.
