MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in
  Language Models, More than Outcomes
As AI systems progress, we rely more on them to make decisions with us and
for us. To ensure that such decisions are aligned with human values, it is
imperative for us to understand not only what decisions they make but also how
they come to those decisions. Reasoning language models, which provide both
final responses and (partially transparent) intermediate thinking traces,
present a timely opportunity to study AI procedural reasoning. Unlike math and
code problems which often have objectively correct answers, moral dilemmas are
an excellent testbed for process-focused evaluation because they allow for
multiple defensible conclusions. To do so, we present MoReBench: 1,000 moral
scenarios, each paired with a set of rubric criteria that experts consider
essential to include (or avoid) when reasoning about the scenarios. MoReBench
contains over 23 thousand criteria including identifying moral considerations,
weighing trade-offs, and giving actionable recommendations to cover cases on AI
advising humans moral decisions as well as making moral decisions autonomously.
Separately, we curate MoReBench-Theory: 150 examples to test whether AI can
reason under five major frameworks in normative ethics. Our results show that
scaling laws and existing benchmarks on math, code, and scientific reasoning
tasks fail to predict models' abilities to perform moral reasoning. Models also
show partiality towards specific moral frameworks (e.g., Benthamite Act
Utilitarianism and Kantian Deontology), which might be side effects of popular
training paradigms. Together, these benchmarks advance process-focused
reasoning evaluation towards safer and more transparent AI.
68f6f4f524c44893631118de
Yu Ying Chiu
68f6f4f524c44893631118df
Michael S. Lee
68f6f4f524c44893631118e0
Rachel Calcott
68f6f4f524c44893631118e1
Brandon Handoko
68f6f4f524c44893631118e2
Paul de Font-Reaulx
68f6f4f524c44893631118e3
Paula Rodriguez
68f6f4f524c44893631118e4
Chen Bo Calvin Zhang
68f6f4f524c44893631118e5
Ziwen Han
68f6f4f524c44893631118e6
Udari Madhushani Sehwag
68f6f4f524c44893631118e7
Yash Maurya
68f6f4f524c44893631118e8
Christina Q Knight
68f6f4f524c44893631118e9
Harry R. Lloyd
68f6f4f524c44893631118ea
Florence Bacus
68f6f4f524c44893631118eb
Mantas Mazeika
68f6f4f524c44893631118ec
Bing Liu
68f6f4f524c44893631118ed
Yejin Choi
68f6f4f524c44893631118ee
Mitchell L Gordon
68f6f4f524c44893631118ef
Sydney Levine
MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in
  Language Models, More than Outcomes
As AI systems progress, we rely more on them to make decisions with us and
for us. To ensure that such decisions are aligned with human values, it is
imperative for us to understand not only what decisions they make but also how
they come to those decisions. Reasoning language models, which provide both
final responses and (partially transparent) intermediate thinking traces,
present a timely opportunity to study AI procedural reasoning. Unlike math and
code problems which often have objectively correct answers, moral dilemmas are
an excellent testbed for process-focused evaluation because they allow for
multiple defensible conclusions. To do so, we present MoReBench: 1,000 moral
scenarios, each paired with a set of rubric criteria that experts consider
essential to include (or avoid) when reasoning about the scenarios. MoReBench
contains over 23 thousand criteria including identifying moral considerations,
weighing trade-offs, and giving actionable recommendations to cover cases on AI
advising humans moral decisions as well as making moral decisions autonomously.
Separately, we curate MoReBench-Theory: 150 examples to test whether AI can
reason under five major frameworks in normative ethics. Our results show that
scaling laws and existing benchmarks on math, code, and scientific reasoning
tasks fail to predict models' abilities to perform moral reasoning. Models also
show partiality towards specific moral frameworks (e.g., Benthamite Act
Utilitarianism and Kantian Deontology), which might be side effects of popular
training paradigms. Together, these benchmarks advance process-focused
reasoning evaluation towards safer and more transparent AI.
MoReBench and MoReBench-Theory provide benchmarks for evaluating AI's moral reasoning and decision-making processes, highlighting the need for process-focused evaluation and transparency in AI systems.
reasoning language models moral dilemmas moral scenarios rubric criteria moral considerations trade-offs actionable recommendations normative ethics Benthamite Act Utilitarianism Kantian Deontology scaling laws math code scientific reasoning process-focused reasoning evaluation